profile
viewpoint
James Munns jamesmunns Ferrous Systems GmbH Berlin, DE ferrous-systems.com Bringing Rust to new places, big and small. Managing Director @ Ferrous Systems GmbH, Core Team @ Rust Embedded Working Group

jamesmunns/awesome-rust-streaming 309

A community curated list of Rust Language streamers

jamesmunns/bbqueue 118

A SPSC, lockless, no_std, thread safe, queue, based on BipBuffers

Covertness/coap-rs 107

A Constrained Application Protocol(CoAP) library implemented in Rust.

http-rs/async-h1 97

Asynchronous HTTP/1.1 in Rust

http-rs/http-types 75

Common types for HTTP operations

iqlusioninc/usbarmory.rs 41

Bare metal Rust support for USB armory MkII devices

braun-embedded/rust-ieee802.15.4 24

Rust implementation of the IEEE 802.15.4 standard

braun-embedded/rust-dw1000 21

Rust driver crate for the Decawave DW1000 UWB transceiver

braun-embedded/rust-dwm1001 15

[DEPRECATED] This repository was merged with https://github.com/braun-embedded/rust-dw1000

jamesmunns/bluepill 6

Board Support Package in Rust for the STM32F103 "Blue Pill" board

push eventjamesmunns/home-fleet

James Munns

commit sha d791464e0c2e75fc3b1ace52f70c76df682de3fc

Expose pipe

view details

James Munns

commit sha c5d3ed983aa5fa024de892ed08e505d02faeef32

Work from the stream

view details

push time in 5 hours

issue commentrust-lang/rust

`#[inline]` prevents symbol from appearing in `staticlib` output

Expanding on @Aaron1011's note, I would also expect this not to apply to cdylibs as well (I used the staticlib for my example, but my use case generates code for both static and dynamic library usage.

jamesmunns

comment created time in 2 days

PR opened probe-rs/probe-rs-rtt

Shorten lock duration
+4 -3

0 comment

1 changed file

pr created time in 3 days

create barnchjamesmunns/probe-rs-rtt

branch : lock-reduce

created branch time in 3 days

fork jamesmunns/probe-rs-rtt

Library for interfacing with the RTT (Real-Time Transfer) debugging protocol over probe-rs

fork in 3 days

push eventjamesmunns/home-fleet

James Munns

commit sha 2b8f1c46810d8d92f520216de30d57749da4e148

Update to released version of esb, cargo fmt

view details

push time in 3 days

push eventjamesmunns/home-fleet

James Munns

commit sha 899d3b415ee139f88472e6d07e4ca1cdd991f2f9

Mostly implemented PTX

view details

James Munns

commit sha a3c7f8c2f9414285d4bf0d7da0fdc8ca612b1f13

Work

view details

James Munns

commit sha eed8cc6109f134daf1bc0fc738aa009b5797f4b0

Increase timeout for longer responses

view details

push time in 4 days

push eventjamesmunns/esb

James Munns

commit sha fdcd3885de65894ce85efe64048050708260c9fb

Fix formatting

view details

push time in 4 days

PR opened thalesfragoso/esb

Provide write access to a read frame
+12 -1

0 comment

2 changed files

pr created time in 4 days

create barnchjamesmunns/esb

branch : mut-read

created branch time in 4 days

created tagjamesmunns/bbqueue

tagv0.4.8

A SPSC, lockless, no_std, thread safe, queue, based on BipBuffers

created time in 4 days

push eventjamesmunns/bbqueue

James Munns

commit sha 38d28a0df2ecb812fac90e31565ff2875696b3dc

0.4.8 release

view details

push time in 4 days

push eventjamesmunns/bbqueue

James Munns

commit sha 521c89babf131d30f98648658297f1994e56f298

Allow write access to a read grant (#64) * Impl DerefMut for read grants * Add a buf_mut method too

view details

push time in 4 days

PR opened jamesmunns/bbqueue

Allow write access to a read grant
+32 -4

0 comment

2 changed files

pr created time in 4 days

create barnchjamesmunns/bbqueue

branch : write-read

created branch time in 4 days

push eventjamesmunns/home-fleet

James Munns

commit sha 4490f721f5c714866b27d327d91849031185144a

Start building radio abstraction

view details

push time in 5 days

create barnchjamesmunns/esb-demos

branch : 840-work

created branch time in 5 days

create barnchjamesmunns/home-fleet

branch : master

created branch time in 5 days

created repositoryjamesmunns/home-fleet

Devices on my home network

created time in 5 days

pull request commentrust-embedded/discovery

Added contextual sections on memory mapping and peripherals

r? me

Tagging so I remember to check this, @adamgreig feel free to take it back if you get to it before me :)

markhildreth

comment created time in 7 days

issue commentzulip/zulip

Add support for denying guest users access to all other users in the organization

Speaking specifically, depending on the cost, we could be interested in sponsoring the implementation of this feature.

We really love using Zulip, and would prefer not to move to another platform for comms outside of our organization, or to have to juggle 5-10 Zulip organizations at once.

timabbott

comment created time in 9 days

issue commentzulip/zulip

Add support for denying guest users access to all other users in the organization

Hey there @timabbott, our company would like to use guests in this kind of way (they can only see folks that share streams with them). Is there anything as a customer/business we can do to help prioritize this feature?

Alternatively, if you know any way of making it easier to exist in multiple zulip organizations at once (some kind of multi-organization web interface), that could be another mitigation for us.

We tend to have customer interactions, and having them be part of our chat would be a huge workflow win for us, but due to NDAs and such, we can't generally have them be visible to other guests.

timabbott

comment created time in 9 days

push eventjamesmunns/keywing-rs

James Munns

commit sha 563efa0d02de05a2f3a35ea2a9d689df4c8cd55b

Remove dead code

view details

push time in 9 days

push eventjamesmunns/keywing-rs

James Munns

commit sha fdaa0cd104dcde008b93d7dadb830508084c45c3

Working little text editor

view details

push time in 9 days

push eventjamesmunns/keywing-rs

James Munns

commit sha f0c087b3c6464eab7d19a3a7b8c8835aff3cc6c1

Add a readme

view details

push time in 9 days

create barnchjamesmunns/keywing-rs

branch : master

created branch time in 9 days

created repositoryjamesmunns/keywing-rs

A playground for the Keyboard Featherwing in Rust

created time in 9 days

create barnchjamesmunns/ili9341-rs

branch : dma-hack

created branch time in 9 days

PR opened nrf-rs/nrf-hal

Add impls for spim3 on nrf52840

This could also be implemented for the 52833. CC @blueluna and #148

+6 -0

0 comment

1 changed file

pr created time in 9 days

create barnchjamesmunns/nrf-hal

branch : implement-spim3

created branch time in 9 days

fork jamesmunns/nrf-hal

A Rust HAL for the nRF family of devices

fork in 9 days

fork jamesmunns/ili9341-rs

A WIP, no_std, generic driver for the ILI9341 (and ILI9340C) TFT LCD display

fork in 9 days

PR opened rust-embedded/wg

Reviewers
Add notes from may

@therealprof has the missing notes for 2020-05-12

+170 -0

0 comment

3 changed files

pr created time in 9 days

create barnchrust-embedded/wg

branch : may-notes

created branch time in 9 days

issue commentrust-embedded/wg

All Hands 2020: Draft Agenda Items

All hands didn't happen this year. Closing.

jamesmunns

comment created time in 9 days

issue closedrust-embedded/wg

All Hands 2020: Draft Agenda Items

Placeholder

closed time in 9 days

jamesmunns

issue openednrf-rs/nrf-hal

Make sure all embedded-hal interfaces can implicitly copy

For compat reasons, we should support implicit copy (if the user passes in a slice from .text) of chunks on all interfaces. We currently do this for UARTE, but not for SPIM/TWIM.

created time in 11 days

issue openednrf-rs/nrf-hal

Rework 840 SPIM support

While working on the 840, I realized:

  • We don't implement SPIM3
  • SPIM0-SPIM2 only support a max speed of 8MHz. If you use a higher speed, it is oddly slow
  • We don't do implicit copies for embedded-hal

I should be able to submit fixes for this, though I'm not sure if any of the other targets support SPIM3 as well.

created time in 11 days

create barnchjamesmunns/cc13x0-pac-demo

branch : master

created branch time in 13 days

created repositoryjamesmunns/cc13x0-pac-example

created time in 13 days

created repositoryjamesmunns/cc13x0-pac

created time in 13 days

issue commentrust-embedded/cortex-m-rt

Add warning about `use pac`

CC #268

jamesmunns

comment created time in 13 days

issue closedrust-embedded/cortex-m-rt

Add warning about `use pac`

If you don't use the pac, just include it as a dependency, you get a confusing "missing device feature" warning. This bit me.

I'll add a warning to link.x.in after my stream :)

closed time in 13 days

jamesmunns

issue commentrust-embedded/cortex-m-rt

Add warning about `use pac`

Just kidding, this exists in the unreleased version here: https://github.com/rust-embedded/cortex-m-rt/blob/master/link.x.in#L194

jamesmunns

comment created time in 13 days

issue openedrust-embedded/cortex-m-rt

Add warning about `use pac`

If you don't use the pac, just include it as a dependency, you get a confusing "missing device feature" warning. This bit me.

I'll add a warning to link.x.in after my stream :)

created time in 13 days

issue openedrust-lang/rust

`#[inline]` prevents symbol from appearing in `staticlib` output

I currently use Rust to generate a static library for FFI consumption. In some public interfaces, I call other FFI public interfaces as primarily a wrapper with a few tweaks. I'd like to mark these "inner" functions as inline, however, this seems to cause them to not show up in the final staticlib.

Given this drastically simplified example:

/// lib.rs
#![no_std]

#[no_mangle]
pub extern "C" fn square_plus_one(input: usize) -> usize {
    square(input) + 1
}

#[no_mangle]
#[inline]
pub extern "C" fn square(input: usize) -> usize {
    input * input
}

use core::panic::PanicInfo;

/// This function is called on panic.
#[panic_handler]
fn panic(_info: &PanicInfo) -> ! {
    loop {
        // Prevent loop mis-optimization
        continue;
    }
}
# Cargo.toml
[package]
name = "inline-ffi"
version = "0.2.0"
edition = "2018"
description = ""
license = ""
documentation = ""
homepage = ""
repository = ""

[lib]
crate-type = ["staticlib", "cdylib"]

[profile.dev]
opt-level = 1
debug = true
lto = true
incremental = false
codegen-units = 1
panic = "abort"

[profile.release]
opt-level = 3
debug = true
lto = true
incremental = false
codegen-units = 1
panic = "abort"

The square_plus_one function makes it to the static library, but the square function does not:

nm -nSC ./target/release/libinline_ffi.a | rg square
0000000000000000 0000000000000009 T square_plus_one

If I remove the #[inline] attribute, we see both functions in the static lib:

nm -nSC ./target/release/libinline_ffi.a | rg square
0000000000000000 0000000000000008 T square
0000000000000000 0000000000000009 T square_plus_one

The relevant docs don't seem to hint that #[inline] can prevent #[no_mangle]/pub extern "C" items from surviving all the way to the final library, or that they shouldn't be used together.

I would expect any function marked pub extern "C" and/or #[no_mangle] should be present in the final artifact, regardless of #[inline] attributes.

I see this behavior on 1.43.1 stable, as well as a recent nightly. I don't know if this is a regression, as this is the first time I've run into this.

Thanks!

created time in 14 days

pull request commentrust-embedded/wg

Hibernating myself

Removed from cortex-m, HAL, and resources GH teams.

thejpster

comment created time in 16 days

pull request commentrust-embedded/wg

Hibernating myself

bors r+

thejpster

comment created time in 16 days

PR opened arturo182/bbq10kbd_i2c_sw

Add published Rust driver
+1 -0

0 comment

1 changed file

pr created time in 17 days

push eventjamesmunns/bbq10kbd_i2c_sw

James Munns

commit sha b43efb1ea5d5e345c4703174a2d490fb24aa373c

Add published Rust driver

view details

push time in 17 days

created tagjamesmunns/bbq10kbd

tagv0.1.0

An Embedded-HAL driver for @arturo182's bbq10kbd PMOD

created time in 17 days

push eventjamesmunns/bbq10kbd

James Munns

commit sha 4a4177c8a810ca4c3476255972725290741f6f89

Update docs

view details

James Munns

commit sha 5900ad89a6221e93702278a5fadc8dd2b65f0479

Save docs

view details

push time in 17 days

create barnchjamesmunns/bbq10kbd

branch : master

created branch time in 17 days

created repositoryjamesmunns/bbq10kbd

An Embedded-HAL driver for @arturo182's bbq10kbd PMOD

created time in 17 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.+A lot of internal data structures are heap-allocated `Arc`-droplets, we overuse hashing and underuse indexing, we don't even intern identifiers!++To get a feeling of how blazingly fast compiler front-ends can be, I highly recommend checking out Sorbet, type checker for Ruby.+You can start with these two links:+* https://blog.nelhage.com/post/why-sorbet-is-fast/+* https://www.youtube.com/watch?v=Gdx6by6tcvw++I am very inspired by this work, but also embarrassed by how far rust-analyzer is from that kind of raw performance and simplicity.++Part of that I think is essential complexity -- Rust's name resolution and macro expansion are *hard*.+But I also wonder if we can change salsa, such that it uses `Vec`-based arenas, rather than `Arc`s in `HashMap`s.++== Parallel and Fast > Persistence++One of the current peculiarities of rust-analyzer is that it doesn't persist caches to disk.+Opening project in rust-analyzer means waiting a dozen seconds while we process standard library and dependencies.++I think this "limitation" is actually a very valuable asset!+It forces us to keep non-incremental code-path reasonably fast.++I think it is plausible that we don't actually need persistent caches at all.+rust-analyzer is basically text processing, and the size of input is in tens of megabytes (_and_ we ignore most of those megabytes anyway).+If we just don't lose performance here and there, and throw the work onto all the cores, we should be able to load projects from scratch withing a reasonable time budget.++The first step here would be establishing the culture of continuous benchmarking and performance tuning.++We've already successfully used rust-analyzer for figuring an architecture which works in IDE at all.+Now it's time to experiment with architecture which works, _fast_, just as all Rust code should :-)++== Optimizing Build Times++In my opinion the two important characteristics that determine long-term success of a project are:++* How long does it take to execute most of the tests?+* How long does it take to build a release version of the project for testing?++I am very happy with test speed in rust-analyzer.+One of my mistakes in IntelliJ is adding a lot of tests that use Rust standard library and are slow for that reason.+In rust-analyzer, there's only three uber-integrated tests that need the real libstd, all others work from in-memory fixtures which contain only the relevant bits of std.++But the build times leave a lot to be desired.+And this is hugely important -- the faster you can build the code, the faster you can do everything else.+Heck, even for improving build times you need fast build times!+I was trying to do some compile-time optimizations in rust-analyzer recently, and measuring "is it faster to compile now?" takes a lot of time, so one has to try fewer different optimizations!++The biggest problem here is that Rust, as a language, is hard to compile fast.+One specific issue I hit constantly is that changing a deep dependency recompiles the world.+This is in contrast to C/C++ where, if you don't touch a `.h` files, changing a dependency requires only re-linking.+In theory, we can have something like this in Rust, by automatically deriving public headers from crates.+Though I fear that without explicit, physical "this is ABI" boundary, it will be much less effective at keeping compile times under control.++As an aside, if Rust stuck with `.crate` files, implementing IDE support would have been much easier :-)++== Optimizing rustc Build++Nonetheless, rust-analyzer is much easier to build than rustc.+I believe there's a lot we can do for rustc build as well.++I've written at length about this at length on https://internals.rust-lang.org/t/experience-report-contributing-to-rust-lang-rust/12012/17?u=matklad[irlo].+The gist is that I think we can split rustc into front-end "text-processing" part, and backend "LLVM, linkers and real world" part.+The front bit than could in theory be a bog standard Rust project, which doesn't depend on IO, external tools and C++ code at all.++One wrinkle here is that rustc test suite at the moment consists predominantly of UI and run-pass tests integration, which work by building the whole compiler.+Such test suite is ideal for testing conformance and catching regressions, but is not really well suited for rapid TDD.+I think we should make an effort to build a unit test suite a-la rust-analyzer, such that it's easy, for example, to test name resolution without building the type checker, and which doesn't require stdlib.++== Scaling Maintainance++Finally, all changes here represent deep cuts into an existing body of software.+Pushing such ambitious projects to completion require people, who can dedicate significant amounts of their time and energy.+To put it bluntly, we need more dedicated folks working on the IDE tooling as a full time, payed job.
To put it bluntly, we need more dedicated folks working on the IDE tooling as a full time, paid job.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.+A lot of internal data structures are heap-allocated `Arc`-droplets, we overuse hashing and underuse indexing, we don't even intern identifiers!++To get a feeling of how blazingly fast compiler front-ends can be, I highly recommend checking out Sorbet, type checker for Ruby.+You can start with these two links:+* https://blog.nelhage.com/post/why-sorbet-is-fast/+* https://www.youtube.com/watch?v=Gdx6by6tcvw++I am very inspired by this work, but also embarrassed by how far rust-analyzer is from that kind of raw performance and simplicity.++Part of that I think is essential complexity -- Rust's name resolution and macro expansion are *hard*.+But I also wonder if we can change salsa, such that it uses `Vec`-based arenas, rather than `Arc`s in `HashMap`s.++== Parallel and Fast > Persistence++One of the current peculiarities of rust-analyzer is that it doesn't persist caches to disk.+Opening project in rust-analyzer means waiting a dozen seconds while we process standard library and dependencies.++I think this "limitation" is actually a very valuable asset!+It forces us to keep non-incremental code-path reasonably fast.++I think it is plausible that we don't actually need persistent caches at all.+rust-analyzer is basically text processing, and the size of input is in tens of megabytes (_and_ we ignore most of those megabytes anyway).+If we just don't lose performance here and there, and throw the work onto all the cores, we should be able to load projects from scratch withing a reasonable time budget.++The first step here would be establishing the culture of continuous benchmarking and performance tuning.++We've already successfully used rust-analyzer for figuring an architecture which works in IDE at all.+Now it's time to experiment with architecture which works, _fast_, just as all Rust code should :-)++== Optimizing Build Times++In my opinion the two important characteristics that determine long-term success of a project are:++* How long does it take to execute most of the tests?+* How long does it take to build a release version of the project for testing?++I am very happy with test speed in rust-analyzer.+One of my mistakes in IntelliJ is adding a lot of tests that use Rust standard library and are slow for that reason.+In rust-analyzer, there's only three uber-integrated tests that need the real libstd, all others work from in-memory fixtures which contain only the relevant bits of std.++But the build times leave a lot to be desired.+And this is hugely important -- the faster you can build the code, the faster you can do everything else.+Heck, even for improving build times you need fast build times!+I was trying to do some compile-time optimizations in rust-analyzer recently, and measuring "is it faster to compile now?" takes a lot of time, so one has to try fewer different optimizations!++The biggest problem here is that Rust, as a language, is hard to compile fast.+One specific issue I hit constantly is that changing a deep dependency recompiles the world.+This is in contrast to C/C++ where, if you don't touch a `.h` files, changing a dependency requires only re-linking.+In theory, we can have something like this in Rust, by automatically deriving public headers from crates.+Though I fear that without explicit, physical "this is ABI" boundary, it will be much less effective at keeping compile times under control.++As an aside, if Rust stuck with `.crate` files, implementing IDE support would have been much easier :-)++== Optimizing rustc Build++Nonetheless, rust-analyzer is much easier to build than rustc.+I believe there's a lot we can do for rustc build as well.++I've written at length about this at length on https://internals.rust-lang.org/t/experience-report-contributing-to-rust-lang-rust/12012/17?u=matklad[irlo].+The gist is that I think we can split rustc into front-end "text-processing" part, and backend "LLVM, linkers and real world" part.+The front bit than could in theory be a bog standard Rust project, which doesn't depend on IO, external tools and C++ code at all.++One wrinkle here is that rustc test suite at the moment consists predominantly of UI and run-pass tests integration, which work by building the whole compiler.+Such test suite is ideal for testing conformance and catching regressions, but is not really well suited for rapid TDD.
Such a test suite is ideal for testing conformance and catching regressions, but is not really well suited for rapid TDD.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.+A lot of internal data structures are heap-allocated `Arc`-droplets, we overuse hashing and underuse indexing, we don't even intern identifiers!++To get a feeling of how blazingly fast compiler front-ends can be, I highly recommend checking out Sorbet, type checker for Ruby.+You can start with these two links:+* https://blog.nelhage.com/post/why-sorbet-is-fast/+* https://www.youtube.com/watch?v=Gdx6by6tcvw++I am very inspired by this work, but also embarrassed by how far rust-analyzer is from that kind of raw performance and simplicity.++Part of that I think is essential complexity -- Rust's name resolution and macro expansion are *hard*.+But I also wonder if we can change salsa, such that it uses `Vec`-based arenas, rather than `Arc`s in `HashMap`s.++== Parallel and Fast > Persistence++One of the current peculiarities of rust-analyzer is that it doesn't persist caches to disk.+Opening project in rust-analyzer means waiting a dozen seconds while we process standard library and dependencies.++I think this "limitation" is actually a very valuable asset!+It forces us to keep non-incremental code-path reasonably fast.++I think it is plausible that we don't actually need persistent caches at all.+rust-analyzer is basically text processing, and the size of input is in tens of megabytes (_and_ we ignore most of those megabytes anyway).+If we just don't lose performance here and there, and throw the work onto all the cores, we should be able to load projects from scratch withing a reasonable time budget.++The first step here would be establishing the culture of continuous benchmarking and performance tuning.++We've already successfully used rust-analyzer for figuring an architecture which works in IDE at all.+Now it's time to experiment with architecture which works, _fast_, just as all Rust code should :-)++== Optimizing Build Times++In my opinion the two important characteristics that determine long-term success of a project are:++* How long does it take to execute most of the tests?+* How long does it take to build a release version of the project for testing?++I am very happy with test speed in rust-analyzer.+One of my mistakes in IntelliJ is adding a lot of tests that use Rust standard library and are slow for that reason.+In rust-analyzer, there's only three uber-integrated tests that need the real libstd, all others work from in-memory fixtures which contain only the relevant bits of std.++But the build times leave a lot to be desired.+And this is hugely important -- the faster you can build the code, the faster you can do everything else.+Heck, even for improving build times you need fast build times!+I was trying to do some compile-time optimizations in rust-analyzer recently, and measuring "is it faster to compile now?" takes a lot of time, so one has to try fewer different optimizations!++The biggest problem here is that Rust, as a language, is hard to compile fast.+One specific issue I hit constantly is that changing a deep dependency recompiles the world.+This is in contrast to C/C++ where, if you don't touch a `.h` files, changing a dependency requires only re-linking.+In theory, we can have something like this in Rust, by automatically deriving public headers from crates.+Though I fear that without explicit, physical "this is ABI" boundary, it will be much less effective at keeping compile times under control.++As an aside, if Rust stuck with `.crate` files, implementing IDE support would have been much easier :-)++== Optimizing rustc Build++Nonetheless, rust-analyzer is much easier to build than rustc.+I believe there's a lot we can do for rustc build as well.++I've written at length about this at length on https://internals.rust-lang.org/t/experience-report-contributing-to-rust-lang-rust/12012/17?u=matklad[irlo].+The gist is that I think we can split rustc into front-end "text-processing" part, and backend "LLVM, linkers and real world" part.+The front bit than could in theory be a bog standard Rust project, which doesn't depend on IO, external tools and C++ code at all.
The front bit then could, in theory, be a bog standard Rust project, which doesn't depend on IO, external tools or C++ code at all.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.+A lot of internal data structures are heap-allocated `Arc`-droplets, we overuse hashing and underuse indexing, we don't even intern identifiers!++To get a feeling of how blazingly fast compiler front-ends can be, I highly recommend checking out Sorbet, type checker for Ruby.+You can start with these two links:+* https://blog.nelhage.com/post/why-sorbet-is-fast/+* https://www.youtube.com/watch?v=Gdx6by6tcvw++I am very inspired by this work, but also embarrassed by how far rust-analyzer is from that kind of raw performance and simplicity.++Part of that I think is essential complexity -- Rust's name resolution and macro expansion are *hard*.+But I also wonder if we can change salsa, such that it uses `Vec`-based arenas, rather than `Arc`s in `HashMap`s.++== Parallel and Fast > Persistence++One of the current peculiarities of rust-analyzer is that it doesn't persist caches to disk.+Opening project in rust-analyzer means waiting a dozen seconds while we process standard library and dependencies.++I think this "limitation" is actually a very valuable asset!+It forces us to keep non-incremental code-path reasonably fast.++I think it is plausible that we don't actually need persistent caches at all.+rust-analyzer is basically text processing, and the size of input is in tens of megabytes (_and_ we ignore most of those megabytes anyway).+If we just don't lose performance here and there, and throw the work onto all the cores, we should be able to load projects from scratch withing a reasonable time budget.++The first step here would be establishing the culture of continuous benchmarking and performance tuning.++We've already successfully used rust-analyzer for figuring an architecture which works in IDE at all.+Now it's time to experiment with architecture which works, _fast_, just as all Rust code should :-)++== Optimizing Build Times++In my opinion the two important characteristics that determine long-term success of a project are:++* How long does it take to execute most of the tests?+* How long does it take to build a release version of the project for testing?++I am very happy with test speed in rust-analyzer.+One of my mistakes in IntelliJ is adding a lot of tests that use Rust standard library and are slow for that reason.+In rust-analyzer, there's only three uber-integrated tests that need the real libstd, all others work from in-memory fixtures which contain only the relevant bits of std.++But the build times leave a lot to be desired.+And this is hugely important -- the faster you can build the code, the faster you can do everything else.+Heck, even for improving build times you need fast build times!+I was trying to do some compile-time optimizations in rust-analyzer recently, and measuring "is it faster to compile now?" takes a lot of time, so one has to try fewer different optimizations!++The biggest problem here is that Rust, as a language, is hard to compile fast.+One specific issue I hit constantly is that changing a deep dependency recompiles the world.+This is in contrast to C/C++ where, if you don't touch a `.h` files, changing a dependency requires only re-linking.+In theory, we can have something like this in Rust, by automatically deriving public headers from crates.+Though I fear that without explicit, physical "this is ABI" boundary, it will be much less effective at keeping compile times under control.++As an aside, if Rust stuck with `.crate` files, implementing IDE support would have been much easier :-)++== Optimizing rustc Build++Nonetheless, rust-analyzer is much easier to build than rustc.+I believe there's a lot we can do for rustc build as well.++I've written at length about this at length on https://internals.rust-lang.org/t/experience-report-contributing-to-rust-lang-rust/12012/17?u=matklad[irlo].+The gist is that I think we can split rustc into front-end "text-processing" part, and backend "LLVM, linkers and real world" part.
The gist is that I think we can split rustc into a front-end "text-processing" part, and backend "LLVM, linkers and real world" part.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.+A lot of internal data structures are heap-allocated `Arc`-droplets, we overuse hashing and underuse indexing, we don't even intern identifiers!++To get a feeling of how blazingly fast compiler front-ends can be, I highly recommend checking out Sorbet, type checker for Ruby.+You can start with these two links:+* https://blog.nelhage.com/post/why-sorbet-is-fast/+* https://www.youtube.com/watch?v=Gdx6by6tcvw++I am very inspired by this work, but also embarrassed by how far rust-analyzer is from that kind of raw performance and simplicity.++Part of that I think is essential complexity -- Rust's name resolution and macro expansion are *hard*.+But I also wonder if we can change salsa, such that it uses `Vec`-based arenas, rather than `Arc`s in `HashMap`s.++== Parallel and Fast > Persistence++One of the current peculiarities of rust-analyzer is that it doesn't persist caches to disk.+Opening project in rust-analyzer means waiting a dozen seconds while we process standard library and dependencies.++I think this "limitation" is actually a very valuable asset!+It forces us to keep non-incremental code-path reasonably fast.++I think it is plausible that we don't actually need persistent caches at all.+rust-analyzer is basically text processing, and the size of input is in tens of megabytes (_and_ we ignore most of those megabytes anyway).+If we just don't lose performance here and there, and throw the work onto all the cores, we should be able to load projects from scratch withing a reasonable time budget.++The first step here would be establishing the culture of continuous benchmarking and performance tuning.++We've already successfully used rust-analyzer for figuring an architecture which works in IDE at all.+Now it's time to experiment with architecture which works, _fast_, just as all Rust code should :-)++== Optimizing Build Times++In my opinion the two important characteristics that determine long-term success of a project are:++* How long does it take to execute most of the tests?+* How long does it take to build a release version of the project for testing?++I am very happy with test speed in rust-analyzer.+One of my mistakes in IntelliJ is adding a lot of tests that use Rust standard library and are slow for that reason.+In rust-analyzer, there's only three uber-integrated tests that need the real libstd, all others work from in-memory fixtures which contain only the relevant bits of std.++But the build times leave a lot to be desired.+And this is hugely important -- the faster you can build the code, the faster you can do everything else.+Heck, even for improving build times you need fast build times!+I was trying to do some compile-time optimizations in rust-analyzer recently, and measuring "is it faster to compile now?" takes a lot of time, so one has to try fewer different optimizations!++The biggest problem here is that Rust, as a language, is hard to compile fast.+One specific issue I hit constantly is that changing a deep dependency recompiles the world.+This is in contrast to C/C++ where, if you don't touch a `.h` files, changing a dependency requires only re-linking.+In theory, we can have something like this in Rust, by automatically deriving public headers from crates.+Though I fear that without explicit, physical "this is ABI" boundary, it will be much less effective at keeping compile times under control.++As an aside, if Rust stuck with `.crate` files, implementing IDE support would have been much easier :-)++== Optimizing rustc Build++Nonetheless, rust-analyzer is much easier to build than rustc.+I believe there's a lot we can do for rustc build as well.++I've written at length about this at length on https://internals.rust-lang.org/t/experience-report-contributing-to-rust-lang-rust/12012/17?u=matklad[irlo].
I've written at length about this on https://internals.rust-lang.org/t/experience-report-contributing-to-rust-lang-rust/12012/17?u=matklad[irlo].
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.+A lot of internal data structures are heap-allocated `Arc`-droplets, we overuse hashing and underuse indexing, we don't even intern identifiers!++To get a feeling of how blazingly fast compiler front-ends can be, I highly recommend checking out Sorbet, type checker for Ruby.+You can start with these two links:+* https://blog.nelhage.com/post/why-sorbet-is-fast/+* https://www.youtube.com/watch?v=Gdx6by6tcvw++I am very inspired by this work, but also embarrassed by how far rust-analyzer is from that kind of raw performance and simplicity.++Part of that I think is essential complexity -- Rust's name resolution and macro expansion are *hard*.+But I also wonder if we can change salsa, such that it uses `Vec`-based arenas, rather than `Arc`s in `HashMap`s.++== Parallel and Fast > Persistence++One of the current peculiarities of rust-analyzer is that it doesn't persist caches to disk.+Opening project in rust-analyzer means waiting a dozen seconds while we process standard library and dependencies.++I think this "limitation" is actually a very valuable asset!+It forces us to keep non-incremental code-path reasonably fast.++I think it is plausible that we don't actually need persistent caches at all.+rust-analyzer is basically text processing, and the size of input is in tens of megabytes (_and_ we ignore most of those megabytes anyway).+If we just don't lose performance here and there, and throw the work onto all the cores, we should be able to load projects from scratch withing a reasonable time budget.++The first step here would be establishing the culture of continuous benchmarking and performance tuning.++We've already successfully used rust-analyzer for figuring an architecture which works in IDE at all.+Now it's time to experiment with architecture which works, _fast_, just as all Rust code should :-)++== Optimizing Build Times++In my opinion the two important characteristics that determine long-term success of a project are:++* How long does it take to execute most of the tests?+* How long does it take to build a release version of the project for testing?++I am very happy with test speed in rust-analyzer.+One of my mistakes in IntelliJ is adding a lot of tests that use Rust standard library and are slow for that reason.+In rust-analyzer, there's only three uber-integrated tests that need the real libstd, all others work from in-memory fixtures which contain only the relevant bits of std.++But the build times leave a lot to be desired.+And this is hugely important -- the faster you can build the code, the faster you can do everything else.+Heck, even for improving build times you need fast build times!+I was trying to do some compile-time optimizations in rust-analyzer recently, and measuring "is it faster to compile now?" takes a lot of time, so one has to try fewer different optimizations!++The biggest problem here is that Rust, as a language, is hard to compile fast.+One specific issue I hit constantly is that changing a deep dependency recompiles the world.+This is in contrast to C/C++ where, if you don't touch a `.h` files, changing a dependency requires only re-linking.
This is in contrast to C/C++ where, if you don't touch any `.h` files, changing a dependency requires only re-linking.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.+A lot of internal data structures are heap-allocated `Arc`-droplets, we overuse hashing and underuse indexing, we don't even intern identifiers!++To get a feeling of how blazingly fast compiler front-ends can be, I highly recommend checking out Sorbet, type checker for Ruby.+You can start with these two links:+* https://blog.nelhage.com/post/why-sorbet-is-fast/+* https://www.youtube.com/watch?v=Gdx6by6tcvw++I am very inspired by this work, but also embarrassed by how far rust-analyzer is from that kind of raw performance and simplicity.++Part of that I think is essential complexity -- Rust's name resolution and macro expansion are *hard*.+But I also wonder if we can change salsa, such that it uses `Vec`-based arenas, rather than `Arc`s in `HashMap`s.++== Parallel and Fast > Persistence++One of the current peculiarities of rust-analyzer is that it doesn't persist caches to disk.+Opening project in rust-analyzer means waiting a dozen seconds while we process standard library and dependencies.++I think this "limitation" is actually a very valuable asset!+It forces us to keep non-incremental code-path reasonably fast.++I think it is plausible that we don't actually need persistent caches at all.+rust-analyzer is basically text processing, and the size of input is in tens of megabytes (_and_ we ignore most of those megabytes anyway).+If we just don't lose performance here and there, and throw the work onto all the cores, we should be able to load projects from scratch withing a reasonable time budget.++The first step here would be establishing the culture of continuous benchmarking and performance tuning.++We've already successfully used rust-analyzer for figuring an architecture which works in IDE at all.+Now it's time to experiment with architecture which works, _fast_, just as all Rust code should :-)++== Optimizing Build Times++In my opinion the two important characteristics that determine long-term success of a project are:++* How long does it take to execute most of the tests?+* How long does it take to build a release version of the project for testing?++I am very happy with test speed in rust-analyzer.+One of my mistakes in IntelliJ is adding a lot of tests that use Rust standard library and are slow for that reason.+In rust-analyzer, there's only three uber-integrated tests that need the real libstd, all others work from in-memory fixtures which contain only the relevant bits of std.
In rust-analyzer, there are only three uber-integrated tests that need the real libstd, all others work from in-memory fixtures which contain only the relevant bits of std.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.+A lot of internal data structures are heap-allocated `Arc`-droplets, we overuse hashing and underuse indexing, we don't even intern identifiers!++To get a feeling of how blazingly fast compiler front-ends can be, I highly recommend checking out Sorbet, type checker for Ruby.+You can start with these two links:+* https://blog.nelhage.com/post/why-sorbet-is-fast/+* https://www.youtube.com/watch?v=Gdx6by6tcvw++I am very inspired by this work, but also embarrassed by how far rust-analyzer is from that kind of raw performance and simplicity.++Part of that I think is essential complexity -- Rust's name resolution and macro expansion are *hard*.+But I also wonder if we can change salsa, such that it uses `Vec`-based arenas, rather than `Arc`s in `HashMap`s.++== Parallel and Fast > Persistence++One of the current peculiarities of rust-analyzer is that it doesn't persist caches to disk.+Opening project in rust-analyzer means waiting a dozen seconds while we process standard library and dependencies.++I think this "limitation" is actually a very valuable asset!+It forces us to keep non-incremental code-path reasonably fast.++I think it is plausible that we don't actually need persistent caches at all.+rust-analyzer is basically text processing, and the size of input is in tens of megabytes (_and_ we ignore most of those megabytes anyway).+If we just don't lose performance here and there, and throw the work onto all the cores, we should be able to load projects from scratch withing a reasonable time budget.++The first step here would be establishing the culture of continuous benchmarking and performance tuning.++We've already successfully used rust-analyzer for figuring an architecture which works in IDE at all.+Now it's time to experiment with architecture which works, _fast_, just as all Rust code should :-)++== Optimizing Build Times++In my opinion the two important characteristics that determine long-term success of a project are:++* How long does it take to execute most of the tests?+* How long does it take to build a release version of the project for testing?++I am very happy with test speed in rust-analyzer.+One of my mistakes in IntelliJ is adding a lot of tests that use Rust standard library and are slow for that reason.
One of my mistakes in IntelliJ is adding a lot of tests that use Rust's standard library and are slow for that reason.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.+A lot of internal data structures are heap-allocated `Arc`-droplets, we overuse hashing and underuse indexing, we don't even intern identifiers!++To get a feeling of how blazingly fast compiler front-ends can be, I highly recommend checking out Sorbet, type checker for Ruby.+You can start with these two links:+* https://blog.nelhage.com/post/why-sorbet-is-fast/+* https://www.youtube.com/watch?v=Gdx6by6tcvw++I am very inspired by this work, but also embarrassed by how far rust-analyzer is from that kind of raw performance and simplicity.++Part of that I think is essential complexity -- Rust's name resolution and macro expansion are *hard*.+But I also wonder if we can change salsa, such that it uses `Vec`-based arenas, rather than `Arc`s in `HashMap`s.++== Parallel and Fast > Persistence++One of the current peculiarities of rust-analyzer is that it doesn't persist caches to disk.+Opening project in rust-analyzer means waiting a dozen seconds while we process standard library and dependencies.++I think this "limitation" is actually a very valuable asset!+It forces us to keep non-incremental code-path reasonably fast.++I think it is plausible that we don't actually need persistent caches at all.+rust-analyzer is basically text processing, and the size of input is in tens of megabytes (_and_ we ignore most of those megabytes anyway).+If we just don't lose performance here and there, and throw the work onto all the cores, we should be able to load projects from scratch withing a reasonable time budget.++The first step here would be establishing the culture of continuous benchmarking and performance tuning.++We've already successfully used rust-analyzer for figuring an architecture which works in IDE at all.+Now it's time to experiment with architecture which works, _fast_, just as all Rust code should :-)++== Optimizing Build Times++In my opinion the two important characteristics that determine long-term success of a project are:++* How long does it take to execute most of the tests?+* How long does it take to build a release version of the project for testing?++I am very happy with test speed in rust-analyzer.
I am very happy with the testing speed of rust-analyzer.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.+A lot of internal data structures are heap-allocated `Arc`-droplets, we overuse hashing and underuse indexing, we don't even intern identifiers!++To get a feeling of how blazingly fast compiler front-ends can be, I highly recommend checking out Sorbet, type checker for Ruby.+You can start with these two links:+* https://blog.nelhage.com/post/why-sorbet-is-fast/+* https://www.youtube.com/watch?v=Gdx6by6tcvw++I am very inspired by this work, but also embarrassed by how far rust-analyzer is from that kind of raw performance and simplicity.++Part of that I think is essential complexity -- Rust's name resolution and macro expansion are *hard*.+But I also wonder if we can change salsa, such that it uses `Vec`-based arenas, rather than `Arc`s in `HashMap`s.++== Parallel and Fast > Persistence++One of the current peculiarities of rust-analyzer is that it doesn't persist caches to disk.+Opening project in rust-analyzer means waiting a dozen seconds while we process standard library and dependencies.++I think this "limitation" is actually a very valuable asset!+It forces us to keep non-incremental code-path reasonably fast.++I think it is plausible that we don't actually need persistent caches at all.+rust-analyzer is basically text processing, and the size of input is in tens of megabytes (_and_ we ignore most of those megabytes anyway).+If we just don't lose performance here and there, and throw the work onto all the cores, we should be able to load projects from scratch withing a reasonable time budget.++The first step here would be establishing the culture of continuous benchmarking and performance tuning.++We've already successfully used rust-analyzer for figuring an architecture which works in IDE at all.

This one is a stylistic and typo suggestion :)

matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.+A lot of internal data structures are heap-allocated `Arc`-droplets, we overuse hashing and underuse indexing, we don't even intern identifiers!++To get a feeling of how blazingly fast compiler front-ends can be, I highly recommend checking out Sorbet, type checker for Ruby.+You can start with these two links:+* https://blog.nelhage.com/post/why-sorbet-is-fast/+* https://www.youtube.com/watch?v=Gdx6by6tcvw++I am very inspired by this work, but also embarrassed by how far rust-analyzer is from that kind of raw performance and simplicity.++Part of that I think is essential complexity -- Rust's name resolution and macro expansion are *hard*.+But I also wonder if we can change salsa, such that it uses `Vec`-based arenas, rather than `Arc`s in `HashMap`s.++== Parallel and Fast > Persistence++One of the current peculiarities of rust-analyzer is that it doesn't persist caches to disk.+Opening project in rust-analyzer means waiting a dozen seconds while we process standard library and dependencies.++I think this "limitation" is actually a very valuable asset!+It forces us to keep non-incremental code-path reasonably fast.++I think it is plausible that we don't actually need persistent caches at all.+rust-analyzer is basically text processing, and the size of input is in tens of megabytes (_and_ we ignore most of those megabytes anyway).+If we just don't lose performance here and there, and throw the work onto all the cores, we should be able to load projects from scratch withing a reasonable time budget.++The first step here would be establishing the culture of continuous benchmarking and performance tuning.++We've already successfully used rust-analyzer for figuring an architecture which works in IDE at all.
We've already successfully used rust-analyzer for forging an architecture which works in IDE at all.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.+A lot of internal data structures are heap-allocated `Arc`-droplets, we overuse hashing and underuse indexing, we don't even intern identifiers!++To get a feeling of how blazingly fast compiler front-ends can be, I highly recommend checking out Sorbet, type checker for Ruby.+You can start with these two links:+* https://blog.nelhage.com/post/why-sorbet-is-fast/+* https://www.youtube.com/watch?v=Gdx6by6tcvw++I am very inspired by this work, but also embarrassed by how far rust-analyzer is from that kind of raw performance and simplicity.++Part of that I think is essential complexity -- Rust's name resolution and macro expansion are *hard*.+But I also wonder if we can change salsa, such that it uses `Vec`-based arenas, rather than `Arc`s in `HashMap`s.++== Parallel and Fast > Persistence++One of the current peculiarities of rust-analyzer is that it doesn't persist caches to disk.+Opening project in rust-analyzer means waiting a dozen seconds while we process standard library and dependencies.++I think this "limitation" is actually a very valuable asset!+It forces us to keep non-incremental code-path reasonably fast.++I think it is plausible that we don't actually need persistent caches at all.+rust-analyzer is basically text processing, and the size of input is in tens of megabytes (_and_ we ignore most of those megabytes anyway).+If we just don't lose performance here and there, and throw the work onto all the cores, we should be able to load projects from scratch withing a reasonable time budget.
If we just don't lose performance here and there, and throw the work onto all the cores, we should be able to load projects from scratch within a reasonable time budget.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.+A lot of internal data structures are heap-allocated `Arc`-droplets, we overuse hashing and underuse indexing, we don't even intern identifiers!++To get a feeling of how blazingly fast compiler front-ends can be, I highly recommend checking out Sorbet, type checker for Ruby.+You can start with these two links:+* https://blog.nelhage.com/post/why-sorbet-is-fast/+* https://www.youtube.com/watch?v=Gdx6by6tcvw++I am very inspired by this work, but also embarrassed by how far rust-analyzer is from that kind of raw performance and simplicity.++Part of that I think is essential complexity -- Rust's name resolution and macro expansion are *hard*.+But I also wonder if we can change salsa, such that it uses `Vec`-based arenas, rather than `Arc`s in `HashMap`s.++== Parallel and Fast > Persistence++One of the current peculiarities of rust-analyzer is that it doesn't persist caches to disk.+Opening project in rust-analyzer means waiting a dozen seconds while we process standard library and dependencies.++I think this "limitation" is actually a very valuable asset!+It forces us to keep non-incremental code-path reasonably fast.
It forces us to keep the non-incremental code-path reasonably fast.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.+However, I have a lot of doubts about specific details.+I feel that today rust-analyzer lacks mechanical sympathy and leaves a tone of performance on the table.
I feel that today rust-analyzer lacks mechanical sympathy and leaves a ton of performance on the table.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.+Of course, as an IDE author I'd love to require specifying auto-traits, but, as a language user, I much prefer the current design.++== Compact Data Structures++rust-analyzer uses a novel and rather high-tech query-based architecture for incremental computation.+Today, it is clear that this general approach fits IDE use-case really well.
Today, it is clear that this general approach fits an IDE use-case really well.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:++* local inherent impls with publicly visible methods.+* local trait impls for non-local types.+* `#[macro_export]` local macros.+* local out-of-line modules.++If we want to have fast & correct IDE support, we should phase out those from the language via edition mechanism.++Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect function body if you call the function.
Note that auto-trait leakage of impl Trait is not nearly as problematic, as you only need to inspect a function's body if you call the function.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.++Rust mostly confirms to this property, but there are a couple of annoying violations:
Rust mostly conforms to this property, but there are a couple of annoying violations:
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:++__What happens inside a function, stays inside the function__++If it is possible to type-check body of a function without looking at the bodies of other functions, you can speed up IDE by drastically reducing the amount of work it needs to do.
If it is possible to type-check the body of a function without looking at the bodies of other functions, you can speed up an IDE by drastically reducing the amount of work it needs to do.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)+Long term, I believe that guaranteeing deterministic compilation is hugely important, irrespective of IDE story.++== Language Design for Locality++There's a very important language property and IDE can leverage to massively improve performance:
There's a very important language property that an IDE can leverage to massively improve performance:
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,+* it's hard to limit execution time of proc-macro,+* proc-macros can be non-deterministic, which can break internal IDE invariants.++At the moment, we paper over this by running proc-macros in a separate process and never invalidating proc-macro caches, but this feels like a hack and has high implementation complexity.+it would be much better if proc-macros were deterministic and controllable by definition, and WASM can give us that.++I am slightly worried that this will get push-back from folks who want to connect to database over TCP at compile time :)
I am slightly worried that this will get push-back from folks who want to connect to databases over TCP at compile time :)
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.+This works ok-enough for the compiler, but is a pretty bad fit for an IDE:++* if proc-macro crashes, it brings down the whole process,
* if a proc-macro crashes, it brings down the whole process,
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.+However, specifying a set of globs up-front makes changing this set later messy.++I would be curious to hear about existing solutions in this area.+One specific question I have is: "How does watchman handle dynamic addition/removal of projects?".+If you have any experience to share, please comment on the VFS issue in rust-analyzer.+Ideally, we turn VFS into just a crates.io crate, as it seems generally useful, and can encapsulate quite a bit of complexity.++The current VFS is ... not great, I don't feel comfortable building rust-analyzer on top of it.++== WASM proc macros++At the moment, proc-macros are implemented as dynamic libraries, loadable into compiler process.
At the moment, proc-macros are implemented as dynamic libraries, loadable into the compiler process.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.+It needs to bridge case-insensitive file systems, symlinks and cycles to a simpler model of "tree with utf8 paths" we want inside.+Additionally it should work with non-path files: there are use-cases where we want to do analysis of Rust code, which doesn't really reside on the file system.++One specific aspect I am struggling with is dynamism.+On the one hand, it seems that a good design is to require to specify the set of files in VFS upfront, as a set of globs.+This is important because, to properly register file watchers without loosing updates, it's important to crawl the file-system eagerly.
This is important because, to properly register file watchers without losing updates, it's important to crawl the file-system eagerly.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.+It serves two dual goals:++* providing consistent immutable snapshots of the file system,+* applying transactional changes to the state of the file system.++This abstraction is the boundary between pure-functional universe of rust-analyzer, and the messiness of external world.
This abstraction is the boundary between the pure-functional universe of rust-analyzer, and the messiness of external world.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.++== Bringing Chalk to Rustc++rust-analyzer has been using chalk as its trait solver for a long time now.+It would be good to finish the work, and integrate it into rustc as well, ~~and give people their GATs~~.++== Single Parser and Syntax Tree++We should share the parser between rustc and rust-analyzer already.+Parsing is one of the most interesting bits of the compiler, from the IDE point of view.+By transitioning rustc to a lossless syntax we'll cross the most important barrier, and it will be a downhill road from that point on.+The design space here I think is well-understood, but the amount of work to do is large.+At some point, I should take a break from actively hacking on rust-analyzer and focus on sharing the parser.++== Virtual File System++The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is VFS.
The most fundamental data structure in rust-analyzer, even more fundamental than a syntax tree, is the VFS, or Virtual File System.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.++Separately, I want to make sure that rust-analyzer is usable inside large non-Cargo based monorepos.+We have some initial support for this already, but there's a bunch of details we need to iron out.++== Dangers of Accidental Architecture++The main danger I see is that rust-analyzer can ossify, as it is today.+This would be bad, because, although current rust-analyzer architecture is right in broad strokes, a lot of important and hard-to-change details are wrong.+After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the decade.
After we push rust-analyzer to the general public, we should focus on boring implementation & design work, with relatively few shiny gifs and a lot of foundational work for the next decade.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.+I think our immediate goal is to make rust-analyzer easier to use in its current state, effectively implementing https://github.com/rust-lang/rfcs/pull/2912[RFC2912].++The amount of programming work on rust-analyzer side is relatively small here:+we need to fix various protocol conformance issues,+clean up various defaults to be less experimental,+write documentation which doesn't require a lot of enthusiasm to understand, etc.+The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge RLS and rust-analyzer VS Code extension, figure out repository structure, etc.
The amount of org stuff is much bigger -- we need to package rust-analyzer with rustup, merge the RLS and rust-analyzer VS Code extensions, figure out repository structure, etc.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.++== rust-analyzer as the Official LSP Server for Rust++People write a ton of Rust today, and they deserve at least a baseline IDE support.
People write a ton of Rust today, and they deserve at least a baseline level of IDE support.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.+This creates its own opportunities and dangers, which inform this planing document.
This creates its own opportunities and dangers, which inform this planning document.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.+Obviously, we have massively overshoot this goal: people depend on rust-analyzer for productive Rust programming today.
Obviously, we have massively overshot this goal: people depend on rust-analyzer for productive Rust programming today.
matklad

comment created time in 18 days

Pull request review commentrust-analyzer/rust-analyzer.github.io

rust-analyzer: Next Few Years

+= rust-analyzer: Next Few Years+:sectanchors:+:experimental:+:page-layout: post++rust-analyzer is a new "IDE backend" for the https://www.rust-lang.org/[Rust] programming language.+Support rust-analyzer on https://opencollective.com/rust-analyzer/[Open Collective].++During the past several months, I've been swamped with in-the-trenches rust-analyzer work.+Today, I spontaneously decided to take a step back and think about longer-term "road map" for rust-analyzer.++What follows is my (https://github.com/matklad/[@matklad]) personal thoughts on the matter, they not necessary reflect the consensus view of ide or compiler teams :-)++== Unexpected Success++One of the most surprising aspects of rust-analyzer for me is how useful it already is.+Today, I write Rust code enjoying fast code-completion, mostly correct go to definition and plethora of assists.+Even syntax highlighting inside macros works!++My original plan for rust-analyzer was to write a quick one-two year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.
My original plan for rust-analyzer was to write a quick one-to-two-year hack to demonstrate a proof-of-concept IDE support, something to strive for rather than a finished product.
matklad

comment created time in 18 days

push eventjamesmunns/awesome-rust-streaming

PSeitz

commit sha 686ce94b89076fb751ae2d5a87c2e80eee0cfae8

add pascal seitz (#15) * add pascal seitz * Apply suggestions from code review Co-authored-by: James Munns <james@onevariable.com> * Update README.md Co-authored-by: James Munns <james@onevariable.com>

view details

push time in 19 days

pull request commentjamesmunns/awesome-rust-streaming

add pascal seitz

Thanks!

PSeitz

comment created time in 19 days

push eventjamesmunns/esb

James Munns

commit sha fb4d9fdf16b2c68f66c62c2ea12cf89c9099c399

cargo fmt

view details

push time in 19 days

pull request commentthalesfragoso/esb

WIP to release the buffers

@thalesfragoso this handles the bbqueue side of recombining the handles to allow them to be resplit in potentially a different mode, or just regain access to the owned peripherals.

I think this is the first step towards making PTX and PRX a generic parameter that you can swap between.

jamesmunns

comment created time in 19 days

PR opened thalesfragoso/esb

WIP to release the buffers
+67 -2

0 comment

3 changed files

pr created time in 19 days

create barnchjamesmunns/esb

branch : release-method

created branch time in 19 days

push eventjamesmunns/bbqueue

James Munns

commit sha b148601a94611c25615ff9b722f703367af42496

v0.4.7 release

view details

push time in 19 days

created tagjamesmunns/bbqueue

tagv0.4.7

A SPSC, lockless, no_std, thread safe, queue, based on BipBuffers

created time in 19 days

push eventjamesmunns/bbqueue

James Munns

commit sha ad6c0aca9b788420acfdad7dc28191db80f7aef8

Release methods (#63) * Hmm, this prevents re-entrancy, but I don't think that's necessary Though, we DO need to make sure we have OUR producer and consumer * Add release methods * Clarify doc comments * Fix copy paste error. Thanks @adamgreig!

view details

push time in 19 days

PR merged jamesmunns/bbqueue

Release methods
+133 -0

1 comment

2 changed files

jamesmunns

pr closed time in 19 days

push eventjamesmunns/bbqueue

James Munns

commit sha ca7ed83444b94338095c6eb604a63cfe2e78517d

Fix copy paste error. Thanks @adamgreig!

view details

push time in 19 days

Pull request review commentjamesmunns/bbqueue

Release methods

 where         let (producer, consumer) = self.try_split()?;         Ok((FrameProducer { producer }, FrameConsumer { consumer }))     }++    /// Attempt to release the Producer and Consumer+    ///+    /// This re-initializes the buffer so it may be split in a different mode at a later+    /// time. There must be no read or write grants active, or an error will be returned.+    ///+    /// The `Producer` and `Consumer` must be from THIS `BBBuffer`, or an error will+    /// be returned.+    ///+    /// ```rust+    /// # // bbqueue test shim!+    /// # fn bbqtest() {+    /// use bbqueue::{BBBuffer, consts::*};+    ///+    /// // Create and split a new buffer+    /// let buffer: BBBuffer<U6> = BBBuffer::new();+    /// let (prod, cons) = buffer.try_split().unwrap();+    ///+    /// // Not possible to split twice+    /// assert!(buffer.try_split().is_err());+    ///+    /// // Release the producer and consumer+    /// assert!(buffer.try_release(prod, cons).is_ok());+    ///+    /// // Split the buffer in framed mode+    /// let (fprod, fcons) = buffer.try_split_framed().unwrap();+    /// # // bbqueue test shim!+    /// # }+    /// #+    /// # fn main() {+    /// # #[cfg(not(feature = "thumbv6"))]+    /// # bbqtest();+    /// # }+    /// ```+    pub fn try_release(+        &'a self,+        prod: Producer<'a, N>,+        cons: Consumer<'a, N>,+    ) -> CoreResult<(), (Producer<'a, N>, Consumer<'a, N>)> {+        // Note: Re-entrancy is not possible because we require ownership+        // of the producer and consumer, which are not cloneable. We also+        // can assume the buffer has been split, because++        // Are these our producers and consumers?+        let our_prod = prod.bbq.as_ptr() as *const Self == self;+        let our_cons = prod.bbq.as_ptr() as *const Self == self;

I should test this too

jamesmunns

comment created time in 20 days

more