profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/chrisdone/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

AJChapman/formatting 148

Format strings type-safely with combinators

AndrasKovacs/flatparse 70

Fast parsing from bytestrings

chrisdone/chrisdone-xmonad 34

My xmonad configuration.

chrisdone/audit 26

Code auditing mode for Emacs

chrisdone/bdo 26

Do things in the browser from Emacs, namely update the stylesheet (but maybe more later)

chrisdone/ats-examples 21

Examples from Introduction to Programming in ATS

chrisdone/ace 15

Attempto Controlled English parser and printer

chrisdone/caseof 11

A simple way to query constructors, like cases but slightly more concise

chrisdone/asp-mode 8

A simple ASP mode for Emacs which does syntax highlighting and indentation support

delete branch chrisdone/haskell-language-server

delete branch : benchmarking-kirill

delete time in a day

push eventchrisdone/haskell-language-server

Kirill Zaborsky

commit sha 266c9d0e09f557b70cb9b8e69bf0562d1173bdd5

Fix compilation issues

view details

Kirill Zaborsky

commit sha 705971d49bde763f72c8adb94b41d8b004e1a362

Fix imports

view details

Kirill Zaborsky

commit sha e6c4f175e29f5d8ba590fd11e5b364d4f888910b

Read HLS output up to EOF

view details

Chris Done

commit sha 899e17e98e4d7240f749ed6f72afb12a50802b31

Merge pull request #2 from chrisdone/benchmarking-kirill Benchmarking kirill

view details

push time in a day

pull request commentchrisdone/haskell-language-server

Benchmarking kirill

Cool :+1: This dumps the -s output, presumably.

qrilka

comment created time in a day

push eventchrisdone/haskell-language-server

Chris Done

commit sha 9311607081c497f28f25e93d83644f0ce41d7282

Send shutdown

view details

push time in 3 days

push eventchrisdone/haskell-language-server

Chris Done

commit sha 9c77ec5931a0c3b8a6e9d2e356f6a18990db56ce

Wait for initialLoad

view details

Chris Done

commit sha ca1f29a85d1e350752d6e80d9da9a65cbc1c67c6

Wait for initial and then terminate

view details

push time in 3 days

push eventchrisdone/haskell-language-server

Chris Done

commit sha d60bebdb22699ae4cf97e2f1a9432f59af9acbfe

Add didchange, didopen, don't close stdin

view details

push time in 3 days

push eventchrisdone/proclog

Chris Done

commit sha a809db757518f8f07f6c75ef9428bfefd9d5c6b3

stdout/stderr/stdin files

view details

push time in 3 days

Pull request review commentchrisdone/haskell-language-server

Optimizations

 packages:  ghc-options:   "$everything": -haddock+  "shake": -fsimpl-tick-factor=10000

Right, with profiling I had to set -O0.

chrisdone

comment created time in 4 days

PullRequestReviewEvent

push eventchrisdone/haskell-language-server

Chris Done

commit sha 6ca067a81dac00cde41c0af35fe1e70effae354f

Add Mem

view details

push time in 4 days

Pull request review commentchrisdone/haskell-language-server

Optimizations

 test-suite wrapper-test   hs-source-dirs:     test/wrapper   main-is:            Main.hs   ghc-options:        -Wall++benchmark bench-vscode+  type:               exitcode-stdio-1.0+  main-is:          Main.hs+  hs-source-dirs:   bench-vscode

Oops. Pushed!

chrisdone

comment created time in 4 days

PullRequestReviewEvent

push eventchrisdone/haskell-language-server

Chris Done

commit sha 031a74fd1a76fd0e8a6cd3b9374e25a39f272f19

Add bench-vscode

view details

push time in 4 days

issue commenthaskell/haskell-language-server

Excessive memory usage in monorepos with dozens/hundreds of cabal packages or modules

I think .hie files should be retained in heap/shake graph only for "files of interest" (files that are currently open in the editor). They are loaded at startup for indexing, but once they are indexed they should not be in memory anymore.

We'll try to fix this too.

We're sure that not type-checking the whole project on startup will help initially, but once you start working with the project, it'll have to type check anyway. So it doesn't sound like a big win. But we'll test it anyway.

jneira

comment created time in 4 days

push eventchrisdone/proclog

Chris Done

commit sha cb81403c6c109e0deac6cd3ea4fc24b5b9a5ee13

Update README.md

view details

push time in 4 days

issue commenthaskell/haskell-language-server

Excessive memory usage in monorepos with dozens/hundreds of cabal packages or modules

If you plan to do further analysis, I suggest that you use the #2060 branch

Thanks, we'll use that one!

jneira

comment created time in 4 days

MemberEvent

create barnchchrisdone/haskell-language-server

branch : benchmarking

created branch time in 4 days

push eventchrisdone/proclog

Chris Done

commit sha c53c7004118cf530ecd21ccee501c79bd4d01c52

Add args

view details

push time in 5 days

issue commenthaskell/haskell-language-server

Excessive memory usage in monorepos with dozens/hundreds of cabal packages or modules

shake built with -O0 (I don't think this should matter much) Politely WTF?

It doesn't make a difference to the memory use, reproducing the same 3GB with -O enabled. Stands to reason, shake itself doesn't do much in the way of allocations.

jneira

comment created time in 5 days

push eventchrisdone/proclog

Chris Done

commit sha e3a0a755acf104228563f69e5514a12fe64894dd

Fix

view details

push time in 9 days

push eventchrisdone/proclog

Chris Done

commit sha c6e070fb38d8bdfb708f701a7effca80de2e4c0c

Fix lines

view details

push time in 9 days

issue commenthaskell/haskell-language-server

Excessive memory usage in monorepos with dozens/hundreds of cabal packages or modules

Here are the same numbers with default plugins enabled without any flags passed. The maximum residency doubles to 4GB with total memory use jumping to an almost double 9.7GB.

298,006,080,712 bytes allocated in the heap
   24,459,460,216 bytes copied during GC
    4,421,180,736 bytes maximum residency (11 sample(s))
       24,941,248 bytes maximum slop
             9744 MiB total memory in use (0 MB lost due to fragmentation)
 
                                      Tot time (elapsed)  Avg pause  Max pause
   Gen  0       309 colls,   309 par   66.564s   8.939s     0.0289s    0.1255s
   Gen  1        11 colls,    10 par   18.143s   8.081s     0.7347s    5.2873s
 
   Parallel GC work balance: 86.99% (serial 0%, perfect 100%)
 
   TASKS: 42 (1 bound, 41 peak workers (41 total), using -N8)
 
   SPARKS: 6737 (6649 converted, 0 overflowed, 0 dud, 0 GC'd, 88 fizzled)
 
   INIT    time    0.002s  (  0.002s elapsed)
   MUT     time  421.495s  (242.495s elapsed)
   GC      time   84.707s  ( 17.020s elapsed)
   EXIT    time    5.252s  (  0.003s elapsed)

   Total   time  511.457s  (259.520s elapsed)
 
   Alloc rate    707,021,073 bytes per MUT second
 
   Productivity  82.4% of total user, 93.4% of total elapsed
 

Therefore the summary is: the initial loading has very high memory use, the loading of the .hie files has very high memory use, and the plugins have very high memory use.


As a small point of interest, the memory use before we consult the cradle in ghcide leads us already to 4GB maximum residency and 1GB total use.

2021-09-16 22:46:58.56733376 UTC stderr> 2021-09-16 23:46:58.567153867 [ThreadId 108] INFO hls:	Consulting the cradle for "app/main.hs"
 Killing self before calling loadCradle.
2021-09-16 22:46:58.574672845 UTC stderr> Received SIGTERM. Killing main thread.
2021-09-16 22:46:58.575577644 UTC stderr> Output from setting up the cradle Cradle {cradleRootDir = "/home/chris/Work/mercury/web-backend", cradleOptsProg = CradleAction: Stack}
2021-09-16 22:46:58.575869799 UTC stderr> haskell-language-server: thread killed
2021-09-16 22:46:58.602032681 UTC stderr>      
      157,480,464 bytes allocated in the heap
        1,445,800 bytes copied during GC
        4,188,152 bytes maximum residency (1 sample(s))
          178,184 bytes maximum slop
             1045 MiB total memory in use (0 MB lost due to fragmentation)
 
                                      Tot time (elapsed)  Avg pause  Max pause
   Gen  0         0 colls,     0 par    0.000s   0.000s     0.0000s    0.0000s
   Gen  1         1 colls,     0 par    0.019s   0.019s     0.0187s    0.0187s
 
   TASKS: 31 (1 bound, 27 peak workers (30 total), using -N8)
 
   SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)
 
   INIT    time    0.003s  (  0.003s elapsed)
   MUT     time    0.208s  (  0.792s elapsed)
   GC      time    0.019s  (  0.019s elapsed)
   EXIT    time    0.005s  (  0.007s elapsed)
   Total   time    0.235s  (  0.820s elapsed)
 
   Alloc rate    756,162,690 bytes per MUT second
 
   Productivity  88.8% of total user, 96.5% of total elapsed

At this point we have some easy areas to investigate.

I'm likely to proceed with heap/allocation profiling tools from here to identify in each stage if there are any low-hanging fruit.

jneira

comment created time in 9 days

issue commenthaskell/haskell-language-server

Excessive memory usage in monorepos with dozens/hundreds of cabal packages or modules

Some initial observations on the same codebase @parsonsmatt is talking about. We're working on improving HLS's performance on this codebase.

The (private) codebase has about 1401 modules in one single package.

Configuration:

  • I have disabled all plugins for HLS via stack build --flag haskell-language-server:-all-formatters --flag haskell-language-server:-all-plugins --flag haskell-language-server:-brittany --flag haskell-language-server:-callhierarchy --flag haskell-language-server:-class --flag haskell-language-server:-eval --flag haskell-language-server:-floskell --flag haskell-language-server:-fourmolu --flag haskell-language-server:-haddockcomments --flag haskell-language-server:-hlint --flag haskell-language-server:-importlens --flag haskell-language-server:-modulename --flag haskell-language-server:-ormolu --flag haskell-language-server:-pedantic --flag haskell-language-server:-pragmas --flag haskell-language-server:-refineimports --flag haskell-language-server:-rename --flag haskell-language-server:-retrie --flag haskell-language-server:-splice --flag haskell-language-server:-stylishhaskell --flag haskell-language-server:-tactic --copy-bins
  • shake built with -O0 (I don't think this should matter much)

HLS uses for stack the stack ghci for generating .hie files with args like "-fwrite-ide-info","-hiedir",".hiefiles....

Here is -s output when I hardcode HLS to self-terminate before it begins loading .hie files, with 2.6GB of total memory use or 678MB of maximum residency (56GB total allocations).

56,391,925,584 bytes allocated in the heap
    4,541,399,808 bytes copied during GC
      678,908,224 bytes maximum residency (7 sample(s))
        7,761,600 bytes maximum slop
             2626 MiB total memory in use (0 MB lost due to fragmentation)
 
                                      Tot time (elapsed)  Avg pause  Max pause
   Gen  0        47 colls,    47 par    9.485s   1.318s     0.0280s    0.0475s
   Gen  1         7 colls,     6 par    4.927s   1.324s     0.1892s    0.7458s
 
   Parallel GC work balance: 93.47% (serial 0%, perfect 100%)
 
   TASKS: 35 (1 bound, 34 peak workers (34 total), using -N8)
 
   SPARKS: 21 (20 converted, 0 overflowed, 0 dud, 0 GC'd, 1 fizzled)
 
   INIT    time    0.002s  (  0.002s elapsed)
   MUT     time   60.577s  ( 58.391s elapsed)
   GC      time   14.412s  (  2.642s elapsed)
   EXIT    time    0.005s  (  0.006s elapsed)
   Total   time   74.996s  ( 61.040s elapsed)
 
   Alloc rate    930,911,213 bytes per MUT second
 
   Productivity  80.8% of total user, 95.7% of total elapsed

Meanwhile, if I allow HLS to proceed with loading all the .hie files, then we have 6GB total memory use with a maximum residency of 2GB (and 251GB total allocations).

251,921,600,112 bytes allocated in the heap
   21,768,183,432 bytes copied during GC
    2,288,714,288 bytes maximum residency (11 sample(s))
       16,694,000 bytes maximum slop
             6218 MiB total memory in use (0 MB lost due to fragmentation)
 
                                      Tot time (elapsed)  Avg pause  Max pause
   Gen  0       296 colls,   296 par   53.105s   6.924s     0.0234s    0.1068s
   Gen  1        11 colls,    10 par   17.618s   4.554s     0.4140s    2.1406s
 
   Parallel GC work balance: 84.73% (serial 0%, perfect 100%)
 
   TASKS: 41 (1 bound, 40 peak workers (40 total), using -N8)
 
   SPARKS: 4711 (4676 converted, 0 overflowed, 0 dud, 0 GC'd, 35 fizzled)
 
2021-09-16 21:34:46.35954889 UTC stderr
   INIT    time    0.002s  (  0.002s elapsed)
   MUT     time  391.014s  (224.797s elapsed)
   GC      time   70.723s  ( 11.478s elapsed)
   EXIT    time    0.007s  (  0.003s elapsed)
   Total   time  461.746s  (236.280s elapsed)
 
   Alloc rate    644,276,909 bytes per MUT second
 
   Productivity  84.7% of total user, 95.1% of total elapsed

That's this many .hie files:

$ find /home/chris/.cache/ghcide/main-14b8ce32153e652c2e5dfdbd46a19ccdbb440779 -name '*.hie' | wc -l
		1392

It's hardly surprising, as the HieFile type has a lot of data in it.

So my initial impressions are:

  1. Simply loading up the codebase files causes a lot of allocations and residency is high, even before we load .hie files or think about enabling plugins. Ideally we could shrink that down and start letting go of unneeded things but it may not be the main burden.
  2. Once we load up .hie files, that takes up 3x more memory than even the first step. The HieFile contents is large, and bringing them all into memory is naturally a blow up area. Avoiding loading them up, or loading them and discarding them for later with an IO action that loads them on demand might be an option.

I'll follow up with numbers when plugins are enabled.

jneira

comment created time in 9 days

push eventchrisdone/proclog

Chris Done

commit sha 127aa01b50aaecfb59cbd13a3e33bb3ee945c3a2

Handle sigterm

view details

push time in 9 days

push eventchrisdone/proclog

Chris Done

commit sha 7b92b01ac2e22acbc4ded52c52cbf909979b35f5

Tweak

view details

push time in 9 days

push eventchrisdone/proclog

Chris Done

commit sha f61a2e2b67e508be4ec754ce64d556db0783f110

Document

view details

push time in 9 days

create barnchchrisdone/proclog

branch : master

created branch time in 9 days