profile
viewpoint

graydon/awesome-c 2

A curated list of awesome C frameworks, libraries and software.

graydon/awesome-scalability 1

The Patterns of Scalable, Reliable, and Performant Large-Scale Systems

davidungar/swift 0

The Swift Programming Language

graydon/1ml 0

1ML prototype interpreter

graydon/8086tiny 0

Official repository for 8086tiny: a tiny PC emulator/virtual machine

graydon/adapton.rust 0

Nominal Adapton in Rust

graydon/aliceml 0

A functional programming language based on Standard ML, extended with support for concurrent, distributed, and constraint programming

graydon/alternative-internet 0

A collection of interesting new networks and tech aiming at decentralisation (in some form).

graydon/antidote 0

SyncFree Reference Platform

graydon/art-rs 0

Adaptive Radix Tree in Rust

push eventgraydon/stellar-core

Graydon Hoare

commit sha 41c17bd4a6e86202304e0395463e980633bda689

Add tests for medida percentiles and exp-decay reservoir sample

view details

Graydon Hoare

commit sha eea7d2178c85248e2cfc92e35bca2c25710f3340

bump libmedida

view details

push time in 6 hours

fork graydon/py-tpcc

Python implementation of TPC-C

fork in 6 hours

startedapavlo/py-tpcc

started time in 6 hours

PR opened stellar/stellar-core

Check medida math

Description

This adds some testing machinery to confirm that medida is calculating percentiles properly, and that when it runs an exponential-decay reservoir sampler it's not corrupting the samples too much.

As far as I can tell so far both these statements are mostly true: it seems like it's doing the right thing. Ish. I fixed one small bug in the percentile calculation routine (see https://github.com/stellar/medida/pull/15) which I don't think would have been especially dire in any case.

It's possible I'm not finding the smoking gun yet because, in part, these tests are noisy and I had to add pretty wide margins of error for them to pass reliably. When they fail, they dump out the resulting array of samples and you can copy them into R and ask it for percentiles, and so far I'm not seeing anything out of the ordinary except that (say) gamma functions have long tails and even uniform samples have a relatively high variance when we're only examining 1028 of them. Things narrow out if we have 100,000 samples, but that's .. not what the medida reservoirs have. So.

I'll keep looking tomorrow, but I figured this is worthwhile machinery to have just to check our own hunches and experiences against. Feel free to extend or suggest scenarios you'd like to see.

+315 -1

0 comment

2 changed files

pr created time in 18 hours

create barnchgraydon/stellar-core

branch : check-medida-math

created branch time in 18 hours

PR opened stellar/medida

Correct -- or at least improve? -- quantile calculation

The previous quantile calculation appears to have been interpolating between the wrong pairs of points: values[floor(h)-1] and values[floor(h)], where it should be interpolating between values[floor(h)] and values[floor(h)+1].

Possibly this is not a huge issue, and possibly I'm still doing something wrong at the endpoints (it's not entirely clear to me what the right condition is for giving up on interpolating, or if interpolation should degrade somehow more gracefully at the endpoints) but this version seems like it behaves a little better (& closer to R, which I take as canonical).

+41 -23

0 comment

1 changed file

pr created time in 18 hours

create barnchgraydon/medida

branch : corrected-quantile-calculation

created branch time in 18 hours

delete branch graydon/medida

delete branch : corrected-percentile-calculation

delete time in 18 hours

create barnchgraydon/medida

branch : corrected-percentile-calculation

created branch time in 18 hours

PR opened stellar/go

[WIP] exp/ingest Captive-stellar-core sketch

This is an initial sketch of support for an ingestion backend driven by a captive stellar-core subprocess replaying ledgers in memory. It's going to need a bunch of iteration before it's viable, but this PR should serve as a place to examine & discuss the approach, fix bugs, etc.

+7527 -7011

0 comment

5 changed files

pr created time in a day

push eventgraydon/go

Bartek Nowotarski

commit sha 9bbfc3c81bdaab65bda5dc6d39f19d24e44e636b

support/db: Support concurrent queries in a transaction (#2024) This commit adds `Synchronized` flag to `support/db.Session`. When set to `true` and `Session` runs a transaction all `Exec*` and `Select*` methods are protected by mutex what allows running them in multiple goroutines. This is an experimental feature (see below) and not a breaking change (default is `false`). Postgres protocol does not allow sending Exec query results if previously sent Query haven't been fully read. This issue manifested itself when a PR that's sending read and write queries in multiple goroutines was merged. More info: lib/pq#81 lib/pq#635 Known limitations: * It's possible that it will not be needed if we decide to remove concurrency from ingestion pipeline (see #1983). We are adding this to unblock development of new ingestion processors with a more readable code (currently all database processors are merged into one big processor that is hard to read and test). Splitting `DatabaseProcessor` will be done in a separate PR/PRs. * During Horizon Team meeting we agreed to create a new `SynchronizedSession` struct embedding `db.Session` inside that would not be placed in a shared `support` package. The problem is that `history.Q` embeds `db.Session` inside. Changing this to `db.SessionInterface` requires updating a lot of files that create `history.Q` objects, tests and mocks. Given that we may want to revert this change in the future, the change in this commit seems to be simpler.

view details

Bartek Nowotarski

commit sha 2efdfd9fe55a9dfc6fda61c18449a4fffa424447

Merge branch 'master' into release-horizon-v0.25.0

view details

tamirms

commit sha ecdde84912c9867461fd5ad0b37b1bd42ddb7147

Fix deadlock in experimental ingestion (#2059) The db session used in the ingestion system and pipelines was different from the db session used in the pipeline hooks. When starting the ingestion system, System.Run() creates a transaction for the pipeline and does a select for update when calling historyQ.GetLastLedgerExpIngest(). Eventually, the pipeline preprocessing hook is invoked. preProcessingHook() checks if there is a transaction attached to the db session. Because the db session in the pipeline hook is distinct from the db session used by System.Run(), preProcessingHook() finds there is no transaction attached to the session. preProcessingHook() then creates a new transaction and also calls historyQ.GetLastLedgerExpIngest() which triggers a select for update. Now, preProcessingHook() is stuck because the select for update will only terminate when the other transaction in the System db session is rolled back or committed. To fix the deadlock, the pipeline hook should share the same db session as the other ingestion components.

view details

Bartek Nowotarski

commit sha db7f06e584e2b66c848b1b201c46c60fbf0a9847

Merge pull request #2068 from stellar/master Merge master into release-horizon-v0.25.0

view details

Bartek Nowotarski

commit sha 9d7f8755d41996c558dc6cf6fdf142939337e95a

services/horizon: Batch Upsert Accounts and Trust Lines (#2073) This commit changes `DatabaseProcessor` to insert/update (upsert) accounts and trust lines in batches. In #2004 we added `LedgerEntryChangeCache` that aims to decrease number of DB updates: all the changes connected to a single ledger entry are squashed into just one DB query. Even though it gives a nice performance boost when large number of ops change a small number of ledger entries, it turns out this is not enough. When many ledger entries are changed in a new ledger, DB connection round trip time takes significant percentage of time in overall ledger processing time. The SQL query was borrowed [1] from stellar-core. Known limitations: * This code shouldn't be the final code that is released. The aim of this PR is to deploy it to stg environment to check it's performance. * This is adding upsert queries for accounts and trust lines only. These types are the most common changes in the recent ledgers. The final solution should add batch upsert and batch delete for all ledger entries. * There's a potential to improve the code: autogenerating upsert query for any ledger entry type, better tests. Will be done in a separate PR. * It removes a code that check a number of rows affected by a query. Unfortunately, the performance of the current solution is too bad to keep it. [1] https://github.com/stellar/stellar-core/blob/21469f90da1eacc6845017a520e179afb3772e65/src/ledger/LedgerTxnAccountSQL.cpp#L306-L336

view details

tamirms

commit sha 603bb747f2c5efefff4318578c3f094df5089ec7

services/horizon/internal/expingest: Transaction processor (#2037) Add transaction processor to ingest transactions

view details

Adolfo Builes

commit sha b1e98403f3511cd3dc17f68e153ba17e58c4fc90

services/horizon/internal/expingest: Operations and OperationParticipants processor (#2043) * Add exp_history_operations table and operation batch insert builder. * Add transaction_operation struct. * Implement functions for transaction_operation. * Start operation details. * Update details to support more types. * Add test for StrictReceivePathPayment details. * Add test for manageSellOffer and createPassiveSellOffer details. * Add details test for set options. * Add test for all flags and thresholds details. * Add test for allowTrust and changeTrust. * Add test for bumpSequence and manageBuyOffer. * Add pathPaymentStrictSend test. * Add Order() helper to transaction_operation. * Use index to build transaction. * Use history transaction in operation_batch_insert_builder. * Remove comment, index is 0-based. * Use transaction from kahuna. * Update sql and include transaction in db. * Add mocks for QOperations and OperationBatchInsertBuilder. * Add operations_processor. * Add placeholder for CheckExpOperations. * Implemt CheckExpOperations. * Add failing scenario for TestCheckExpOperations. * Refactor buildTransaction * Add OperationsProcessor to pipeline. * Make transactionOperation private and refactor Add. * Fix comment. * Fix tests. * Remove todo. * Make private transaction_operation_wrapper fields. * Refactor buildLedgerTransaction to take *testing.T instead of *test.T. * Use buildLedgerTransaction. * Move more common code to SetupTest. * Move s.mockLedgerReader.On("Close").Return(nil).Once() to setup. * Move mockQ.On("NewOperationBatchInsertBuilder" to SetupTest. * Validate operations in processor. * Update comment. * Add exp_history_operation_participants. * Add helper function to extract a operation's participants. * Fix rebase conflicts. * Add operation participant batch insert builder. * Remove participant meta. * Move QOperations to history/operation. * Redo bindata. * Add mock for operation participants batch and q_operations. * Simplify OperationParticipantBatchInsertBuilder. * Add operation participants processor. * Handle scenario where entry exist but operationSet is nil. * Use correct batch insert builder. * Add CheckExpOperationParticipants * Move temp mocks to mock_q_participants. * Fix participants_processor_test. * Add TestOperationParticipantsBatchAddFails * Add TestOpeartionBatchAddExecFails * Move repeated mocks to their own function. * Fix rebase error. * Create selectOperationFields. * Add test for CheckExpOperationParticipants. * Add helper method findOperationParticipants. * Simplify CheckExpOperationParticipants. * Fix govet error. * Join migration for history_operations and history_operation_participants. * Extend RemoveExpIngestHistory with new tables. * Fix pipeline hooks test. * Simplify CheckExpTransactions and CheckExpOperations * Fix updates to transactionSet and operationSet * Bundle all participant checks into CheckExpParticipants * Log check errors in operations processor instead of returning them * Fix TestRemoveExpIngestHistory * include checkExpOperationParticipants in participant checks * Redo bindata. Co-authored-by: tamirms <tamirms@gmail.com>

view details

Adolfo Builes

commit sha 4c232f725f5d6079a4e8c04c68466df6a9a21efb

services/horizon/internal: Move transaction operation wrapper to operation processor (#2079) * Move transaction_operation_wrapper. * Add helper file to tests. * Update operation_batch_insert_builder. * Make transaction a valid bump sequence. * Update processor to use new batch insert builder format. * Move transaction_operation_wrapper to operation_processor. * Fix govet issues.

view details

Bartek Nowotarski

commit sha a43ba684972719bbd480f3266ed8a1f7722385ce

services/horizon/expingest: Rewrite System to a state machine (#2085) This PR rewrites `System` to a state machine as described in #1969. The new code is easier to test because it's possible to start test from any state without having to add a bunch of mocked methods to reach the code we want to test. It's also easier to extend with new states. This will be very helpful when catchup states are added for history ingestion.

view details

tamirms

commit sha b947313579669c151cb33f5bbff95ac1b34bd798

Remove dead code from legacy ingestion system (#2090)

view details

Bartek Nowotarski

commit sha 853c0a34df750bd9c72b82dba963075ccf28aee0

services/horizon: Fix tests mocking UpsertTrustLines (#2092) Fixes a but in tests that are mocking `UpsertTrustLines` method. Because all of the ledger entry changes are cached using `LedgerEntryChangeCache` it's possible that the order of trustlines in `UpsertTrustLines` will be different between calls if more than one trust line is updated. `LedgerEntryChangeCache` is using map inside which, when iterating, returns elements in random order.

view details

tamirms

commit sha 3fa1730c95747ea9180f5296fa4a03dc798e4b29

services/horizon/internal/db2: Add exp_history_assets table (#2091) Add a migration for the exp_history_assets table and CreateExpAssets() which creates rows in the exp_history_assets table.

view details

Bartek Nowotarski

commit sha 17c72d1d9fea9a93d84bf6f35486ca5efd703054

services/horizon: verify-range command (#2086) This commit adds `horizon expingest verify-range` command and `exp/ingest.RangeSession`. It also adds `GenesisLedgerStateReader` to that streams master account in a given network to allow ingesting ledger entry changes from ledger 1. In order to ensure that new ingestion system works correctly we plan to reingest pubnet history from the first ledger. In order to achieve this we add `horizon expingest verify-range` command that accepts required two params: `--from` and `--to` that describe the ledger range. When started it: * It ingests ledger entries of `from` ledger (must be a checkpoint ledger). * Ingests ledgers `[from+1, to]`. * If `--verify-state` is set, it runs state verifier on `to` ledger (must be a checkpoint ledger). When merged we can start multiple workers ingesting full history in parallel like: horizon expingest verify-range --from 9983 --to 16383 --verify-state horizon expingest verify-range --from 16383 --to 22783 --verify-state ... Internally, it's using a new `RangeSession` from `exp/ingest` package that process a given range of ledger. `RangeSession` will be also used in `horizon reingest` commands. It's also helpful for finding performance issues as it's easy to reingest a given range with a debug server enabled.

view details

Bartek Nowotarski

commit sha 3b9be93543527601c31b148a16559e5837443ea9

exp/ingest: RangeSession return error when ledger not found (#2098) This commit changes `RangeSession` to return error when ledger not found. Contrary to `LiveSession` that needs to wait for next ledger closed by the network `RangeSession` assumes that all ledgers within a range exist. Instead of waiting it should just return error if ledger is not found.

view details

tamirms

commit sha 491787cbd022bb233338487dbc63f28a0f5daf21

services/horizon/internal/db2: Add exp_history_trades table (#2094) * Add migration for exp_history_trades * Add exp_history_trades related db functions

view details

Bartek Nowotarski

commit sha 92db54a8ee66e68c6c579c29c75c974dbcffc4b2

exp/ingest/io: Add support for ingesting txmeta v2 (#2096) This commit adds support for ingesting `TransactionMeta` when `V=2` and updates XDR files. As explained by @MonsieurNicolas in #1902. > There is some missing meta in protocol version older than version 10. This impacts transactions with one time signers (that get removed after operations in v9 and earlier; and during signature verification in v10 and later). Those changes are invisible in the meta which may confuse certain ingestion processors that keep track of signers on accounts.

view details

Bartek Nowotarski

commit sha 7236520a8e7663a948a30f526440059e67d85f1a

exp/ingest/ledgerbackend: Ensure tx meta is v2 when getting protocol version <10 ledgers (#2099) This commit adds a code that checks if tx meta version is v2 for protocol version <10 ledgers. As explained by @MonsieurNicolas in #1902: > There is some missing meta in protocol version older than version 10. This impacts transactions with one time signers (that get removed after operations in v9 and earlier; and during signature verification in v10 and later). Those changes are invisible in the meta which may confuse certain ingestion processors that keep track of signers on accounts. It's possible that users will use stellar-core without `SUPPORTED_META_VERSION=2` and in such case ingestion system can process state changes incorrectly.

view details

Bartek Nowotarski

commit sha 10480c7bc6b36118c58157a2103bd8399a0829f7

exp/ingest/io: Remove support for TransactionMeta.V=0 (#2101) This commit removes support for ingesting `TransactionMeta.V=0` in `io.LedgerTransaction.GetChanges`. The code generating V0 meta was changed in stellar-core but without upgrading the V param. Some V0 meta has only one `LEDGER_ENTRY_STATE` emitted only when a ledger entry is first modified in a ledger but others add it before every `LEDGER_ENTRY_UPDATE` and `LEDGER_ENTRY_REMOVED`. It looks like the code to parse meta to support both cases should be possible but since transaction meta in protocol version <10 can be incomplete (see [1]) users still need to switch to V2 to be able to parse it. [1] https://github.com/stellar/go/issues/1902 [2] https://github.com/stellar/stellar-core/pull/944 [3] https://github.com/stellar/stellar-core/pull/1441

view details

tamirms

commit sha 33f0b566f4a6d55c2b5cdaecce65217b95a4abe5

services/horizon/internal/expingest/processors: Remove sort on addresses in ParticipantsProcessor (#2103) Previously, we sorted addresses before calling ParticipantsQ.CreateExpAccounts(). The only purpose of the sort was to make it easier to mock the expected arguments of ParticipantsQ.CreateExpAccounts() in the ParticipantsProcessor tests. However, it turns out you can use mock.AnythingOfType() and Assert().ElementsMatch() to assert that a mock was called with a list of arguments without enforcing a specific order. With this functionality we can remove the sort in ParticipantsProcessor.

view details

Adolfo Builes

commit sha f41cc371726677ce6449734006d33e436331b7ff

Effects processor (#2104) * Add exp_history_effects table. * Add EffectBatchInsertBuilder. * Add mock for effect batch insert builder. * Add QEffects interface. * Add function to generate account created effects. * Add effects ingestion for payment. * Use enums in test. * Use table test. * Add helper to keep track of order. * Add ingestion for path payment strict receive effects. * Update sequence to make operationID match Kahuna. * Add ingestion for manage sell offer effects. * Add ingestion for manage buy offer effects. * Add ingestion logic for createPassiveSellOfferEffect. * Ingest setOptions effects. * Add ingestion for changeTrust effects. * Add ingestion for allowTrustEffects. * Fix resultXDR. * Add ingestion for accountMergeEffects. * Ingest inflationEffects. * Ingest manageDataEffects. * Fix GetOperationsForV0. * Add ingestion for bumpSequenceEffects. * Fix ./staticcheck.sh. * Ingest pathPaymentStrictSendEffects. * Remove order and use len(effects) + 1. * Change Effects to effects. * Add struct for effects instead of map[string]interface * Add ProcessLedger to EffectProcessor. * Ignore failed transaction effects. * Add regression test for GetOperationChanges on failed transaction. * Add test for happy path GetOperationChanges. * Add CheckExpOperationEffects. * Compare effects while ingesting. * Add exp_history_effects to RemoveExpIngestHistory. * Rename var. * Use errors.Wrapf. * Clarify use of or in check effects fun. * Check if operationIndex exist instead of using Successful(). * Move expSelectEffect to history/effects.go * Move to next transaction earlier if it is not successful. * Use full-query, no need for a var yet. * Use []byte(string) * Break after trustline change has been found. * Break after we have found the change we are interested in. * Add test for trustline updated and removed. * Add extra test for manageData updated and removed. * Remove address sorting and use mock.Run. * Rename migration. * Update pipeline hook test. * Add migration file. * Only call CheckExpOperationEffects if sequence is higher than 10. * Ad Meta.V2 support to GetOperationChanges. * Refactor err2. * Remove support for TransactionMeta.V=0 in GetOperationChanges.

view details

push time in a day

startedebkalderon/tower-lsp

started time in 2 days

startedTyberiusPrime/nested_intervals

started time in 2 days

startedgoogle/llvm-propeller

started time in 2 days

pull request commentstellar/stellar-core

Update overlay default

r+ d34138a

MonsieurNicolas

comment created time in 2 days

PR opened stellar/stellar-core

TCPPeer: avoid bouncing off ASIO before calling messageSender at first

This is plausibly causing more latency than it's helping in throughput (see ongoing saga of https://github.com/stellar/stellar-core/issues/2441)

+1 -6

0 comment

1 changed file

pr created time in 2 days

create barnchgraydon/stellar-core

branch : avoid-asio-bounce-when-sending-message

created branch time in 2 days

pull request commentstellar/stellar-core

Make batch overlay operation configurable

@latobarita: retry

MonsieurNicolas

comment created time in 3 days

pull request commentstellar/stellar-core

Make batch overlay operation configurable

@latobarita retry

MonsieurNicolas

comment created time in 3 days

issue commentstellar/stellar-core

delays in receiving SCP messages from connected peers

@MonsieurNicolas I approved the PR and will run a simulation test or two using it later; ultimately I think we don't really want these to be exposed configs (they're Yet More Knobs To Turn To Wrong Settings) but for the sake of experiment and gaining confidence / adjusting to the right values I agree it'll make for faster turnaround.

MonsieurNicolas

comment created time in 3 days

fork graydon/arugula

a small C++ lattice library

fork in 3 days

startedjhellerstein/arugula

started time in 3 days

pull request commentstellar/stellar-core

Make batch overlay operation configurable

r+ 55c30d5

MonsieurNicolas

comment created time in 3 days

issue commentstellar/stellar-core

delays in receiving SCP messages from connected peers

I spent the day digging in ASIO's buffered stream class and measuring syscalls, pursuing a simpler hypothesis -- that we're delaying writes too much by spending too long in the new synchronous read-dispatch loop -- and I at least got some half-baked evidence that it might help to limit that (PR https://github.com/stellar/stellar-core/pull/2446 caps the iteration count at 10, which appears to knock about 30% off the p99 of scp.timing.externalized in simulation).

I did some experiments locally with it capped at 1 (i.e. always bouncing off ASIO as before, but still using the increased read buffer size so we don't go to the OS as often, and still doing vectored writes to handle higher load) and I would like to try this in simulation later on but I am out of time tonight.

MonsieurNicolas

comment created time in 4 days

PR opened stellar/stellar-core

Dial down the YieldTimer on synchronous read-dispatch loop

Experimental change to attempt to address #2441 by limiting time limit / iteration count of the synchronous read dispatching loop.

+11 -5

0 comment

1 changed file

pr created time in 4 days

create barnchgraydon/stellar-core

branch : finer-grained-sync-read-dispatch

created branch time in 4 days

startedcfallin/cranelift

started time in 4 days

startedcrossterm-rs/crossterm

started time in 4 days

startedabseil/abseil-cpp

started time in 4 days

startedmarcpaq/b1fipl

started time in 4 days

issue commentstellar/stellar-core

delays in receiving SCP messages from connected peers

Yeah, it's possible -- I saw some of this sort of pattern too when looking up close, though I did also see good aggregate numbers in broader simulations. One thing to keep in mind is that the scatter/gather change got rid of buffered writing altogether (we used to use the asio buffered writer as well as reader; the write side was removed to use scatter/gather) so it might actually be writing worse overall, if we're not batching enough into a scatter/gather set. I will do some experiments on this tomorrow.

MonsieurNicolas

comment created time in 5 days

push eventgraydon/go

Graydon Hoare

commit sha 1af0a259b9232010d0a8dd3528c9ed929aac3a67

WIP: sketch a LedgerBackend that talks to captive stellar-core subprocess.

view details

push time in 6 days

Pull request review commentstellar/stellar-core

Make batch overlay operation configurable

 Config::Config() : NODE_SEED(SecretKey::random())     PEER_AUTHENTICATION_TIMEOUT = 2;     PEER_TIMEOUT = 30;     PEER_STRAGGLER_TIMEOUT = 120;+    MAX_BATCH_READ_DELAY_MS = std::chrono::milliseconds(100);+    MAX_BATCH_READ_COUNT = 1024;+    MAX_BATCH_WRITE_COUNT = 1024;+    MAX_BATCH_WRITE_BYTES = 1 * 1024 * 1024;

Why is the batch write byte count configurable but not the batch read byte count?

MonsieurNicolas

comment created time in 6 days

Pull request review commentstellar/stellar-core

Make batch overlay operation configurable

 PEER_TIMEOUT=30 # time when authenticated. PEER_STRAGGLER_TIMEOUT=120 +# MAX_BATCH_READ_DELAY_MS (Integer) default 100+# How long this server can spend processing reads from a peer at once+MAX_BATCH_READ_DELAY_MS=100

I feel like "delay" is the wrong word here -- it's a maximum "duration" or "period" or "window". Delay implies time spent waiting, whereas this is time spent doing stuff without letting ourselves be interrupted.

MonsieurNicolas

comment created time in 6 days

Pull request review commentstellar/stellar-core

Make batch overlay operation configurable

 TCPPeer::messageSender()     auto now = mApp.getClock().now();     size_t expected_length = 0;     mEnqueueTimeOfLastWrite = mWriteQueue.back().mEnqueuedTime;

If you're going to possibly exit the loop early, you need to ensure mEnqueueTimeOfLastWrite holds the mEnqueuedTime of the last message you pushed a buffer for in that loop. Before this change it was just mWriteQueue.back() because we were going to push all the elements in mWriteQueue, but your change here changes that.

MonsieurNicolas

comment created time in 6 days

Pull request review commentstellar/stellar-core

Make batch overlay operation configurable

 PEER_TIMEOUT=30 # time when authenticated. PEER_STRAGGLER_TIMEOUT=120 +# MAX_BATCH_READ_DELAY_MS (Integer) default 100+# How long this server can spend processing reads from a peer at once+MAX_BATCH_READ_DELAY_MS=100++# MAX_BATCH_READ_COUNT (Integer) default 1024+# How many messages can this server read at once from a peer+MAX_BATCH_READ_COUNT=1024++# MAX_BATCH_WRITE_COUNT (Integer) default 1024+# How many messages can this server send at once to a peer+MAX_BATCH_WRITE_COUNT=1024++# MAX_BATCH_WRITE_BYTES (Integer) default 1048576 (1 Megabyte)+# How many bytes can this server send at once to a peer+MAX_BATCH_WRITE_COUNT=1048576

Typo: you put MAX_BATCH_WRITE_COUNT here twice, and omitted MAX_BATCH_WRITE_BYTES.

MonsieurNicolas

comment created time in 6 days

Pull request review commentstellar/stellar-core

Make batch overlay operation configurable

 TCPPeer::messageSender()     auto now = mApp.getClock().now();     size_t expected_length = 0;     mEnqueueTimeOfLastWrite = mWriteQueue.back().mEnqueuedTime;+    size_t maxQueueSize = mApp.getConfig().MAX_BATCH_WRITE_COUNT;+    size_t const maxTotalBytes = mApp.getConfig().MAX_BATCH_WRITE_BYTES;     for (auto& tsm : mWriteQueue)     {         tsm.mIssuedTime = now;         size_t sz = tsm.mMessage->raw_size();         mWriteBuffers.emplace_back(tsm.mMessage->raw_data(), sz);         expected_length += sz;+        // check if we reached any limit+        if (expected_length > maxTotalBytes)+            break;+        if (--maxQueueSize == 0)

Best to use post-decrement; pre-decrement on an unsigned type underflows at zero

MonsieurNicolas

comment created time in 6 days

push eventgraydon/go

Graydon Hoare

commit sha 9d26677bd8706af1f1f889bdf4b514cda7be3c7f

WIP: sketch a LedgerBackend that talks to captive stellar-core subprocess.

view details

push time in 6 days

push eventgraydon/go

Graydon Hoare

commit sha 2f5e20106ee0de309f183ffe5fa79f41b6d64b79

WIP: sketch a LedgerBackend that talks to captive stellar-core subprocess.

view details

push time in 6 days

push eventgraydon/go

Graydon Hoare

commit sha 919b38bd2ebd264371b50d14c15f16e44ae84aff

WIP: sketch a LedgerBackend that talks to captive stellar-core subprocess.

view details

push time in 7 days

push eventgraydon/go

Leigh McCulloch

commit sha 579a15678e26a01a2957b33cd762a052598e7311

txnbuild: remove superfluous signing key in challenge txn test (#2279) ### What Remove superfluous signing key in challenge transaction test that is used to sign the transaction twice when it only needs to sign the transaction once. ### Why The key is added twice which was probably the result of a copy and paste from another tests where we test the behavior of signing a transaction twice. It's unnecessary to sign it twice here and having it sign the transaction twice is noise that is irrelevant to the scenario being tested in this test.

view details

Leigh McCulloch

commit sha f4e38b51e0368dca9e70eaef4e8c112a372a4290

exp/services/webauth: rename commands package to cmd (#2278) ### What Rename the `exp/services/webauth/internal/commands` package to `exp/services/webauth/cmd`. ### Why The cobra commands for the `webauth` app live in the `internal/commands` package but we have a couple apps that are using the convention to put commands in a top-level `cmd` package. E.g. Horizon, Ticker. This makes `webauth` more consistent with those applications. This idea was discussed here: https://github.com/stellar/go/pull/2267#discussion_r379692099

view details

Leigh McCulloch

commit sha f7c9e85f25a8e998eca1e046121a229de412d9ff

exp/services/recoverysigner: add service (#2267) ### What Add a new incomplete experimental service that a Stellar account can use to delegate anyone possessing a phone number or email address the capability to sign transactions for that account. The service will sign transactions for an account for a user if that user has successfully authenticated with a phone number or email address. The service in its first commit contains just the scaffold and authentication pieces. [SEP-10]: https://github.com/stellar/stellar-protocol/blob/master/ecosystem/sep-0010.md ### Why This service is a first implementation of a new SEP I am working on that defines the mechanisms a client and server can use to share the capability to sign with a server that will allow the account holder to ask the server to sign transactions when authenticated with the account holders phone number or email address. The primary use case of this service is to provide account recovery facilities. If a user loses access to their account the user will be able to authenticate with the server with their phone number or email and ask the server to sign a transaction to add a new signer to the account and to regain control of their account. The service is incomplete, as noted above, but is entering the point where multiple people could start working on it to add the missing components listed above so I want to get it out of a branch and into master. ### Coming Next I've got some changes in separate PRs that will add the following coming soon: - Registering an account telling the server which phone numbers or email addresses are allowed to authenticate for the account, authenticated by a [SEP-10] JWT. - Getting a list of accounts, authenticated by a [SEP-10] JWT or a Firebase JWT (Phone/Email). - Signing a transaction, authenticated by a [SEP-10] JWT or a Firebase JWT (Phone/Email). The service is missing some planned features that'll come later: - Data persistence (e.g. Postgres) - Unique keys for each account encrypted - Metrics - Sufficient logging

view details

Leigh McCulloch

commit sha f9575f17ed7d39636bd59ed3a61cdd415fe5bca4

exp/services/recoverysigner: add account registration endpoint (#2274) ### What Add account registration endpoint. ### Why Accounts will need to register with the recovery signer to be able to recover their account.

view details

Graydon Hoare

commit sha 9a694375db938d472b37c1384356b7d2c15cb04e

Update XDR to include LedgerCloseMeta

view details

Graydon Hoare

commit sha 1548ef5f4ee48fac91ea8239fd378d288d9326bf

Add module reference to go-winio and its dependencies.

view details

Graydon Hoare

commit sha 1f0acd03f139819f31d2ebad49b80d2989069cfe

WIP: sketch a LedgerBackend that talks to captive stellar-core subprocess.

view details

push time in 7 days

push eventgraydon/go

Graydon Hoare

commit sha f4bda3e37993ca96066ecdd31231fbeed9522431

Update XDR to include LedgerCloseMeta

view details

push time in 8 days

fork graydon/gorums

Gorums is a framework for simplifying the design and implementation of fault-tolerant quorum-based protocols.

fork in 8 days

startedrelab/gorums

started time in 8 days

fork graydon/smartmerge

A collection of algorithms for reconfigurable atomic storage.

fork in 8 days

startedrelab/smartmerge

started time in 8 days

fork graydon/Cozette

A bitmap programming font optimized for coziness 💜

fork in 9 days

startedslavfox/Cozette

started time in 9 days

pull request commentstellar/stellar-core

[EXPERIMENT] pull-mode flooding

Likely improvement: flush adverts to all peers after a given buffer-read rather than just the peer the read finished on.

graydon

comment created time in 9 days

push eventgraydon/go

Graydon Hoare

commit sha ba98e7300994e0f5aee917a98c9d89b98f6e8836

WIP: sketch a LedgerBackend that talks to captive stellar-core subprocess.

view details

push time in 11 days

push eventgraydon/go

Graydon Hoare

commit sha 012775a97f43ea0532181396209553bf5dd12598

WIP: sketch a LedgerBackend that talks to captive stellar-core subprocess.

view details

push time in 11 days

create barnchgraydon/go

branch : captive-core-expingest

created branch time in 11 days

startedMaterializeInc/materialize

started time in 13 days

startedkoute/memory-profiler

started time in 14 days

startedsecure-compilation/CompCert

started time in 14 days

startedrust-unofficial/awesome-rust

started time in 14 days

startedmicrosoft/ivy

started time in 14 days

push eventgraydon/stellar-core

Graydon Hoare

commit sha c9274c5a1d0ef686c7e0eda151fea46e9d957f76

WIP: classify FLOOD_{ADVERT,DEMAND} as flood messages.

view details

push time in 14 days

push eventgraydon/stellar-core

Graydon Hoare

commit sha b7b5bff31f9069b4be9f12561f42b1a50074ffe7

WIP: reduce advert hashes from 32-byte SHA256s to 8-byte SipHash24s

view details

push time in 14 days

PR opened stellar/stellar-core

[EXPERIMENT] pull-mode flooding

This is a test branch to conduct some basic exploration of pull-mode flooding in the overlay.

+362 -13

0 comment

17 changed files

pr created time in 15 days

push eventgraydon/stellar-core

MonsieurNicolas

commit sha ec8670910bc474e1f38844c6be20bb2916690208

SCP: log additional information when crashing

view details

MonsieurNicolas

commit sha d8f1d451a93a48fc43ef84dba3c531c9865814cc

SCP: add missing reference to preparedPrime from envelope

view details

MonsieurNicolas

commit sha 7b8b8eed58c4456c98cb402cdd5464f479ab23b8

removed empty file applicationtests.cpp

view details

Latobarita

commit sha c8e65a8587ed1571b6ee0477cde38a00fe0af518

Merge pull request #2429 from MonsieurNicolas/fixMissingReferecenceValueSCP Fix missing reference value scp Reviewed-by: MonsieurNicolas

view details

Graydon Hoare

commit sha 92d7da92fa528cea84bf7129823e18886faf02a0

Move duplicate assetToString helper to util/types.h

view details

Graydon Hoare

commit sha 04d05bd063216de7183ab367e32a19b0df884f39

Add --json flag to dump-xdr subcommand, implement via cereal

view details

Graydon Hoare

commit sha 99bdff49242d2b11e1260a4a6f2fad28dff9ae7e

bump xdrpp submodule to absorb cereal fixes for writing json

view details

Latobarita

commit sha 472f98f1e70d9e120d0befee3554f3f1d5386fa0

Merge pull request #2398 from graydon/dump-xdr-to-json Dump xdr to json Reviewed-by: MonsieurNicolas

view details

MonsieurNicolas

commit sha 3c391d5ed9af2ef2931382595b198aaea8939b11

overlay: fix typo that would cause the floodmap to be invalidated aggressively

view details

Latobarita

commit sha 80fe169cfc365c75f59bf0822f3b2982f373843e

Merge pull request #2431 from MonsieurNicolas/scpFloodFix2 overlay: fix typo that would cause the floodmap to be invalidated aggressively Reviewed-by: graydon

view details

Graydon Hoare

commit sha 1ea79c8610e9cba82b1b35cfea678d2c3876fe1b

Add xdrComputeKeyedHash

view details

Graydon Hoare

commit sha bac04b529c2ba1659a1f542703c43543c23fa7cd

WIP: add advert/demand system to flooding

view details

push time in 15 days

startedhawkw/mycelium

started time in 15 days

startedproglang/ldgv

started time in 15 days

startedtailscale/tailscale

started time in 16 days

pull request commentstellar/stellar-core

Os signposts

Unfortunately having tried this the results are significantly less useful -- all the same-named signposts are consolidated into a single track in the profiler UI, so there's little way to make any visual sense of them if events of different types are intermixed at any reasonable frequency. Presumably the LLVM folks only used this approach because they were timing very coarse grained passes (or else they're not even using it for the profiler UI -- they can be extracted from the system logs too).

graydon

comment created time in 16 days

pull request commentstellar/stellar-core

Dump xdr to json

@MonsieurNicolas this is ready again -- this time with incorporation of David's last round of changes (and a bugfix I had to apply to them in https://github.com/xdrpp/xdrpp/pull/23)

graydon

comment created time in 16 days

push eventgraydon/stellar-core

MonsieurNicolas

commit sha 5707fe7c9ff72a374aa79c099a5bc33fe09e9573

update Catch to 2.11.1

view details

MonsieurNicolas

commit sha ba6bca3c7ee957849f927dcd849c20e56cefbb3c

tests: make rng seed non deterministic by default, fix missing command line support

view details

Latobarita

commit sha 21979637ebbaebcb892be5558e2a77b28d41f039

Merge pull request #2382 from MonsieurNicolas/catchUpdate Update default testing behavior to actually be pseudo random by default Reviewed-by: MonsieurNicolas

view details

MonsieurNicolas

commit sha 4c68225841446012758f5ad45854ec17d9d72fdf

fix pending envelope tests: counter must be positive

view details

MonsieurNicolas

commit sha e597e23beff431e89b4074bc7a9d487617ef14c4

move per slot cleanup to the same location eraseBelow and purgeSlots should happen together

view details

MonsieurNicolas

commit sha f31df633fa4a05751f879706423149709514a0ad

SCP: replace getCurrentState by processCurrentState

view details

MonsieurNicolas

commit sha 5e02356cfba82dd12c98850074d8dbd3b986036e

SCP : replace low/high slot lookup by callback

view details

MonsieurNicolas

commit sha acac94db3bef15a539ca21c514ee5c968a48529e

SCP: allow processCurrentState to always include self

view details

MonsieurNicolas

commit sha bcadfe8ea7fdc690f80708a2e98bc19c98569a2b

avoid duplicate values returned for nomination statements

view details

MonsieurNicolas

commit sha 8ff79efcd5893728e67d1cc168107f4ba168f87d

properly return values referenced by BallotProtocol statements

view details

MonsieurNicolas

commit sha 14d2d399881acf9339555dc885be40e1654848a3

Herder: cleanup PendingEnvelopes

view details

MonsieurNicolas

commit sha 94637a2ff359ea499f8f4ed7e97b61ac46bce2dc

PendingMessages: consolidate writes

view details

MonsieurNicolas

commit sha 88de5b743a9f96dd9587b0daa491e62bc80c5986

herder: only refresh metrics when converting to Json counters don't care when they get updated

view details

MonsieurNicolas

commit sha 552887fa21bffafcfb499730ea74b4b928ccbae5

Add SCPEnvelopeWrapper to SCP code

view details

MonsieurNicolas

commit sha e0f4f3d0e0b29ad37d56cf01e92df1db02c4512b

Herder: attach quorum set and txsets to envelopes

view details

MonsieurNicolas

commit sha ea32447b469f0e501a3c318420901b17b6d4364a

SCP: nomination uses wrapped values

view details

MonsieurNicolas

commit sha 1beaf021e621ba3c370cb7dc7ec2c070d7ecba13

SCP: ballot protocol uses wrapped values

view details

MonsieurNicolas

commit sha b199d47dd1d84814ecabd229313eae04db0e4eec

herder: add a map to track known quorum sets

view details

MonsieurNicolas

commit sha 59dcefeaaef519dd3a11d686e8f84ff3630443f5

herder: add a map to track known txsets

view details

MonsieurNicolas

commit sha 7ade7293907ffa3fb80d43c5795de7dd17cded53

herder: ensure that map learns new value introduced by combineCandidates

view details

push time in 16 days

pull request commentxdrpp/xdrpp

Cereal override

(Discussed with David earlier and he approved merging this)

graydon

comment created time in 16 days

push eventxdrpp/xdrpp

David Mazieres

commit sha d2171a614aac61995fbbd2b598ccad92ef91575d

Add support for cereal_override(ar, t, "fieldname") function

view details

David Mazieres

commit sha 4eb65cc8f1c171d0bd266af0cd7f9db87026a001

oh, might as well forward

view details

Graydon Hoare

commit sha 089d18afc03194718289a0003c30e0334b048fa4

Fix cereal_override to use declval<Archive&>

view details

Graydon Hoare

commit sha ae501492b7eb5a829d0d98f2994ae7c98f629757

Merge pull request #23 from graydon/cereal-override Cereal override

view details

push time in 16 days

PR opened xdrpp/xdrpp

Cereal override

This picks up from David's cereal-override branch with a small (but quite obscure!) fix to use std::declval<Archive&> in the has_cereal_override probe. This is necessary because the signatures of nvp_adapter::apply and xdr::archive take their Archive parameter by non-const, non-rval reference (Archive&), and std::declval<Archive> alone resolves to an Archive&& which won't bind to Archive&. But if we call std::declval<Archive&> it's defined to return an Archive&.

(A lot of stackoverflow, C++ standard scrutiny and trial and error discovered this -- I would never have figured it out alone!)

+56 -6

0 comment

3 changed files

pr created time in 16 days

create barnchgraydon/xdrpp

branch : cereal-override

created branch time in 16 days

pull request commentstellar/stellar-core

Os signposts

Ok, so .. I looked at ways people use this in other contexts to try to evaluate how much I was overstating things (you really can't use anything other than a string literal for the name -- not even a template parameter) and I found that there is a kinda-sorta workaround: you can provide additional varargs after the name that go into the os_log formatting subsystem.So for this, one pattern that kinda works is to give all the signposts the same name and then provide a 2nd argument that's a format string like "event %s" and a 3rd argument that's a captured constant. This is how LLVM does it, I'll see what I can do about adapting it to our code. I want to make sure something useful shows up in the profiler though, since that's the entire point of this. I'll try it and report back.

(On the other hand: I should reiterate that this isn't for performance-critical code, it's for marking up profiles with visual spans to orient the reader. So I continue to think it should probably not be the same decorator type that we're using for other trace timing, or the long-operation timing)

graydon

comment created time in 16 days

startedcirconus-labs/circonusllhist

started time in 16 days

fork graydon/libcircllhist

A C implementation of Circonus log-linear histograms

fork in 16 days

startedcirconus-labs/libcircllhist

started time in 16 days

push eventgraydon/stellar-core

Graydon Hoare

commit sha 114b624a7481cf3f3723b9e3937ac2d4fd49e079

even more wip

view details

push time in 16 days

startedpingcap/awesome-database-learning

started time in 17 days

startedtonyg/syndicate

started time in 17 days

create barnchgraydon/stellar-core

branch : better-flooding

created branch time in 18 days

pull request commentstellar/stellar-core

Scp flood fix

r+ a174ffc

MonsieurNicolas

comment created time in 18 days

push eventgraydon/stellar-core

Graydon Hoare

commit sha a981ff5b05cde171cfbd9c09ba1c90693a61f828

Add Async{Read,Write} metrics to OverlayMetrics

view details

Graydon Hoare

commit sha b0139db0e897806c844f7a3c0dd2f81e7aa8a0c3

Use a larger read buffer and synchronously read as much as possible.

view details

Graydon Hoare

commit sha cba7c0e9c4a4ff1d12c60e28a2ab72b73374c80d

Switch from buffered to vectored async_writes.

view details

Graydon Hoare

commit sha b5749d52b7c96e0882551ffad7bded4444eadf0c

Change straggler logic to use last-enqueued time, not last empty time.

view details

push time in 19 days

push eventgraydon/stellar-core

Graydon Hoare

commit sha a981ff5b05cde171cfbd9c09ba1c90693a61f828

Add Async{Read,Write} metrics to OverlayMetrics

view details

Graydon Hoare

commit sha b0139db0e897806c844f7a3c0dd2f81e7aa8a0c3

Use a larger read buffer and synchronously read as much as possible.

view details

Graydon Hoare

commit sha cba7c0e9c4a4ff1d12c60e28a2ab72b73374c80d

Switch from buffered to vectored async_writes.

view details

Graydon Hoare

commit sha b5749d52b7c96e0882551ffad7bded4444eadf0c

Change straggler logic to use last-enqueued time, not last empty time.

view details

push time in 19 days

push eventgraydon/stellar-core

Marc Power

commit sha 02b118d7e83408a5f259f70b362787cf7450931d

Add support for a VSCode Linux Container

view details

Marc Power

commit sha 148bb1ab60fe345ab42c036e887cba9ae6dfb772

Replace pandoc generated man-page with a static version

view details

MonsieurNicolas

commit sha 11163332129af19a0ed2e61b49f18261045248be

Use vscode account in dev container

view details

MonsieurNicolas

commit sha f63e8bc4eddd811e92d92c49d19df2e659d5eb8a

Add instructions on how to work around shared workspace limitations

view details

MonsieurNicolas

commit sha 8ef5bc440c184a882c4d4d8af29ca80364dff0fb

really remove pandoc dependency

view details

marta-lokhova

commit sha 1c10249b69992e3840e5a9df6797f631766981d8

Ignore any archive write commands in simulation mode

view details

Siddharth Suresh

commit sha 1ef9b45c3c25dde1c9fe114d729cad374b4ef8ce

add overlay survey script

view details

Siddharth Suresh

commit sha ce6050010dddfe90bf665e3a9a33e70a9476383e

PR changes

view details

Latobarita

commit sha 5a8f8b13af5f78704bf0e7b5de4314c16a3639a5

Merge pull request #2412 from sisuresh/addSurveyScript Add overlay survey script Reviewed-by: graydon

view details

Latobarita

commit sha 18d5d5ebc12a5d623b523c114a4ac77ae31964ea

Merge pull request #2394 from MonsieurNicolas/devcontainer Add dev container docker settings Reviewed-by: MonsieurNicolas

view details

Geod24

commit sha 4bfa83c6187b70f2907d10198ef1c8a184eb1048

Remove forward-declaration of stellar::Config from SCP It was originally added as part of PR #2127, and should have been removed as part of the fix for #2152.

view details

Latobarita

commit sha b9b41c566a8f46c2526de224707b91121d263cd1

Merge pull request #2420 from Geod24/fixup-deps Remove forward-declaration of stellar::Config from SCP Reviewed-by: MonsieurNicolas

view details

Latobarita

commit sha 33ca622d6d1b09b67c734b5e7d1b33fa9e4c025a

Merge pull request #2409 from marta-lokhova/disable_publish_in_simulation Ignore any archive write commands in simulation mode Reviewed-by: MonsieurNicolas

view details

marta-lokhova

commit sha 22d02cb13ecbdda1529e9cae4e2247bb1b0ecbf1

Make trace logs more detailed, less verbose

view details

marta-lokhova

commit sha 48977d466767f4cc32e6c7192407eca3f5ca2f14

Add fetch duration metric per items

view details

MonsieurNicolas

commit sha f8cf245f74525adfcf4645d7f4d368153daa7749

allow 0 ops ledger upgrade

view details

MonsieurNicolas

commit sha c45df8050913a826b471241ec8257b4b1bb09c2e

tx ban: allow to ban tx not in pool

view details

MonsieurNicolas

commit sha 579a77805868fa2249066f640ff2f0d66c957704

TransactionFrame: add a method to return the number of operations in a transaction

view details

MonsieurNicolas

commit sha f4ddc9d8f563e29bfca8d7d05bd5d5754f7f3c91

LedgerManager: centralize computation of ledger capacity

view details

MonsieurNicolas

commit sha d62fb643cff02de3fda2c2df3c67fd7182b979e1

transaction queue: automatically ban transactions over quota

view details

push time in 19 days

push eventgraydon/stellar-core

marta-lokhova

commit sha 22d02cb13ecbdda1529e9cae4e2247bb1b0ecbf1

Make trace logs more detailed, less verbose

view details

marta-lokhova

commit sha 48977d466767f4cc32e6c7192407eca3f5ca2f14

Add fetch duration metric per items

view details

MonsieurNicolas

commit sha f8cf245f74525adfcf4645d7f4d368153daa7749

allow 0 ops ledger upgrade

view details

MonsieurNicolas

commit sha c45df8050913a826b471241ec8257b4b1bb09c2e

tx ban: allow to ban tx not in pool

view details

MonsieurNicolas

commit sha 579a77805868fa2249066f640ff2f0d66c957704

TransactionFrame: add a method to return the number of operations in a transaction

view details

MonsieurNicolas

commit sha f4ddc9d8f563e29bfca8d7d05bd5d5754f7f3c91

LedgerManager: centralize computation of ledger capacity

view details

MonsieurNicolas

commit sha d62fb643cff02de3fda2c2df3c67fd7182b979e1

transaction queue: automatically ban transactions over quota

view details

Latobarita

commit sha b6f87670697b041c3b9745fd11dc505ef63641ce

Merge pull request #2422 from MonsieurNicolas/banImprove Ban tx improvement Reviewed-by: MonsieurNicolas

view details

Latobarita

commit sha a2d2402d91ad45f803c56ebc01e69d937fdd895a

Merge pull request #2390 from marta-lokhova/improve_logs_and_metrics Improve logs and metrics Reviewed-by: MonsieurNicolas

view details

Graydon Hoare

commit sha 4fa619f9c3f9f458b0ed55ae23c7e503509f5f76

Add a YieldTimer helper class for starvation avoidance.

view details

Graydon Hoare

commit sha eef4e3d708f906e552bcd10703967185736781b3

Add Async{Read,Write} metrics to OverlayMetrics

view details

Graydon Hoare

commit sha e7628a5c8baa5d2fea0e61709c08cd4b8dc6eac1

Use a larger read buffer and synchronously read as much as possible.

view details

Graydon Hoare

commit sha a8847cef63cad9a2a94d7a34eb36d860f0ff65d5

Switch from buffered to vectored async_writes.

view details

Graydon Hoare

commit sha cf658c0a53130dcec45d9f77cb02ae12cae9a8aa

Bounce off ASIO once when initiating messageSender.

view details

Graydon Hoare

commit sha 81ed930962822ee7fb67c0e274d6fbdd54daa488

Change straggler logic to use last-enqueued time, not last empty time.

view details

Graydon Hoare

commit sha 746ef3c64abe6135aaa0ff5dbd3ca9d2c5eb2be7

TCPPeer: remove a bunch of extraneous uses of 'self'

view details

Graydon Hoare

commit sha 95d2e5ad5cc24a5fb8de19810ad4b3469c69551a

Address review comments.

view details

Graydon Hoare

commit sha 4c4e58e674a5a55693000d89422309be4ae1683b

Reduce read buffers to 256KB.

view details

Graydon Hoare

commit sha 994da7e55e45c867b990516d1dd956209e97f2d5

Use const_buffer not mutable_buffer.

view details

push time in 19 days

push eventgraydon/stellar-core

marta-lokhova

commit sha 1c10249b69992e3840e5a9df6797f631766981d8

Ignore any archive write commands in simulation mode

view details

Geod24

commit sha 4bfa83c6187b70f2907d10198ef1c8a184eb1048

Remove forward-declaration of stellar::Config from SCP It was originally added as part of PR #2127, and should have been removed as part of the fix for #2152.

view details

Latobarita

commit sha b9b41c566a8f46c2526de224707b91121d263cd1

Merge pull request #2420 from Geod24/fixup-deps Remove forward-declaration of stellar::Config from SCP Reviewed-by: MonsieurNicolas

view details

Latobarita

commit sha 33ca622d6d1b09b67c734b5e7d1b33fa9e4c025a

Merge pull request #2409 from marta-lokhova/disable_publish_in_simulation Ignore any archive write commands in simulation mode Reviewed-by: MonsieurNicolas

view details

Graydon Hoare

commit sha 3b5dd67fda2ce58d2d5baf38a5802d1f34711f75

Add a YieldTimer helper class for starvation avoidance.

view details

Graydon Hoare

commit sha 7fb4a0f8a4ddb4e9df88627c40277d6fdf4380a3

Add Async{Read,Write} metrics to OverlayMetrics

view details

Graydon Hoare

commit sha 59a28ee47e32618c5a98a805553f0b4b0b64156a

Use a larger read buffer and synchronously read as much as possible.

view details

Graydon Hoare

commit sha 1e2c9273b886a570c5c7911293fab26ff50c4f61

Switch from buffered to vectored async_writes.

view details

Graydon Hoare

commit sha 328b8dafa98473157d25715180a026385584b745

Bounce off ASIO once when initiating messageSender.

view details

Graydon Hoare

commit sha 1c0119d2715ef2516ea1539c755b4f25d280e7e6

Change straggler logic to use last-enqueued time, not last empty time.

view details

Graydon Hoare

commit sha 79635b8a99efb3d05ceadaf6762a6044469c03d2

TCPPeer: remove a bunch of extraneous uses of 'self'

view details

Graydon Hoare

commit sha e9360f61b5bddf3254c0b843cad7a4b37164f075

Address review comments.

view details

Graydon Hoare

commit sha 647b287efa7ccabd89ab0a7f5124de240c503f0f

Reduce read buffers to 256KB.

view details

Graydon Hoare

commit sha 0746c95453ab95d418cdb49634d3e7e8879d9d42

Use const_buffer not mutable_buffer.

view details

push time in 19 days

push eventgraydon/stellar-core

Graydon Hoare

commit sha 57ba44ad244d97d071331d845ab91a9b64992443

Use const_buffer not mutable_buffer.

view details

push time in 20 days

pull request commentstellar/stellar-core

Os signposts

It strictly requires string literals to work.

I also don't intend this to be for just catching "slow" executions: signposts are useful for marking phases in a profile regardless of whether they're slower than some threshold, just to help you interpret what you're looking at while reading the profile. And conversely I don't want signposts to print to stdout or acquire the elpp lock or whatever. They're supposed to be cheap enough to ignore unless you're profiling.

graydon

comment created time in 20 days

push eventgraydon/stellar-core

Graydon Hoare

commit sha 6b225f6b24f933ab1598f269ba37cf872b090c16

Reduce read buffers to 256KB.

view details

push time in 20 days

push eventgraydon/stellar-core

Graydon Hoare

commit sha decb449f2bc436d40de43e4be93647a1c2912dcc

Address review comments.

view details

push time in 20 days

Pull request review commentstellar/stellar-core

Better network IO buffering

 TCPPeer::startRead()         return;     } +    const size_t HDRSZ = 4;     auto self = static_pointer_cast<TCPPeer>(shared_from_this());      assert(self->mIncomingHeader.size() == 0);      if (Logging::logTrace("Overlay"))         CLOG(TRACE, "Overlay") << "TCPPeer::startRead to " << self->toString(); -    self->mIncomingHeader.resize(4);+    self->mIncomingHeader.resize(HDRSZ);++    // We read large-ish (1MB) buffers of data from TCP which might have quite a+    // few messages in them. We want to digest as many of these _synchronously_+    // as we can before we issue an async_read against ASIO.+    YieldTimer yt(mApp.getClock());+    while (mSocket->in_avail() >= HDRSZ && yt.shouldKeepGoing())+    {+        size_t n = mSocket->read_some(asio::buffer(mIncomingHeader));+        if (n != HDRSZ)+        {+            drop("error during header read_some",+                 Peer::DropDirection::WE_DROPPED_REMOTE,+                 Peer::DropMode::IGNORE_WRITE_QUEUE);+            return;+        }+        size_t length = static_cast<size_t>(getIncomingMsgLength());+        if (length != 0)+        {+            if (mSocket->in_avail() >= length)+            {+                // We can finish reading a full message here synchronously.+                mIncomingBody.resize(length);+                n = mSocket->read_some(asio::buffer(mIncomingBody));+                if (n != length)+                {+                    drop("error during body read_some",

yeah, this is all consolidated now. will post update.

graydon

comment created time in 20 days

Pull request review commentstellar/stellar-core

Better network IO buffering

 TCPPeer::startRead()         return;     } +    const size_t HDRSZ = 4;     auto self = static_pointer_cast<TCPPeer>(shared_from_this());      assert(self->mIncomingHeader.size() == 0);      if (Logging::logTrace("Overlay"))         CLOG(TRACE, "Overlay") << "TCPPeer::startRead to " << self->toString(); -    self->mIncomingHeader.resize(4);+    self->mIncomingHeader.resize(HDRSZ);++    // We read large-ish (1MB) buffers of data from TCP which might have quite a+    // few messages in them. We want to digest as many of these _synchronously_+    // as we can before we issue an async_read against ASIO.+    YieldTimer yt(mApp.getClock());+    while (mSocket->in_avail() >= HDRSZ && yt.shouldKeepGoing())+    {+        size_t n = mSocket->read_some(asio::buffer(mIncomingHeader));+        if (n != HDRSZ)+        {+            drop("error during header read_some",+                 Peer::DropDirection::WE_DROPPED_REMOTE,+                 Peer::DropMode::IGNORE_WRITE_QUEUE);+            return;+        }+        size_t length = static_cast<size_t>(getIncomingMsgLength());+        if (length != 0)+        {+            if (mSocket->in_avail() >= length)+            {+                // We can finish reading a full message here synchronously.+                mIncomingBody.resize(length);+                n = mSocket->read_some(asio::buffer(mIncomingBody));

Yeah, good point. I've pulled out all the common error and non-error steps and it reads much better. Thanks!

graydon

comment created time in 20 days

Pull request review commentstellar/stellar-core

Better network IO buffering

 TCPPeer::startRead()         return;     } +    const size_t HDRSZ = 4;     auto self = static_pointer_cast<TCPPeer>(shared_from_this());      assert(self->mIncomingHeader.size() == 0);      if (Logging::logTrace("Overlay"))         CLOG(TRACE, "Overlay") << "TCPPeer::startRead to " << self->toString(); -    self->mIncomingHeader.resize(4);+    self->mIncomingHeader.resize(HDRSZ);++    // We read large-ish (1MB) buffers of data from TCP which might have quite a+    // few messages in them. We want to digest as many of these _synchronously_+    // as we can before we issue an async_read against ASIO.+    YieldTimer yt(mApp.getClock());+    while (mSocket->in_avail() >= HDRSZ && yt.shouldKeepGoing())+    {+        size_t n = mSocket->read_some(asio::buffer(mIncomingHeader));+        if (n != HDRSZ)

Fair point. I think I've fixed this now. Second pair of eyes still quite welcome. Lots of cases.

graydon

comment created time in 20 days

Pull request review commentstellar/stellar-core

Better network IO buffering

 TCPPeer::startRead()         return;     } +    const size_t HDRSZ = 4;     auto self = static_pointer_cast<TCPPeer>(shared_from_this());      assert(self->mIncomingHeader.size() == 0);      if (Logging::logTrace("Overlay"))         CLOG(TRACE, "Overlay") << "TCPPeer::startRead to " << self->toString(); -    self->mIncomingHeader.resize(4);+    self->mIncomingHeader.resize(HDRSZ);++    // We read large-ish (1MB) buffers of data from TCP which might have quite a+    // few messages in them. We want to digest as many of these _synchronously_+    // as we can before we issue an async_read against ASIO.+    YieldTimer yt(mApp.getClock());+    while (mSocket->in_avail() >= HDRSZ && yt.shouldKeepGoing())+    {+        size_t n = mSocket->read_some(asio::buffer(mIncomingHeader));

fixed.

graydon

comment created time in 20 days

Pull request review commentstellar/stellar-core

Better network IO buffering

 OverlayMetrics::OverlayMetrics(Application& app)           app.getMetrics().NewMeter({"overlay", "message", "read"}, "message"))     , mMessageWrite(           app.getMetrics().NewMeter({"overlay", "message", "write"}, "message"))+    , mAsyncRead(

fixed.

graydon

comment created time in 20 days

pull request commentstellar/stellar-core

Better network IO buffering

I believe this is ready to land now

graydon

comment created time in 21 days

issue commentstellar/stellar-core

too much main-thread blocking / starving / maybe we are using ASIO pretty badly

Another place this general family of issue crops up is in the simulation code, where we used to run multiple applications on the same virtual clock but have (as of https://github.com/stellar/stellar-core/pull/1390) switched to using multiple virtual clocks and a secondary outer advance loop. I think this is probably the wrong approach, and what I'd far prefer to see is a world with a single advancing clock per simulation and (again) a dual-queue structure where the application has a notion of work-units-to-do and the ASIO queue only makes a single call per timeslice to the application to "do a unit of work".

(This would also let us more-faithfully represent the sorts of simulation tasks that the various "crank-some" and "crank-until" helpers are trying to do -- we could accurately differentiate waiting for a barrier at the application level from waiting for a barrier at the IO level)

graydon

comment created time in 21 days

push eventgraydon/stellar-core

Graydon Hoare

commit sha 985fb4197dddc3b45481f38e375cae627d7084de

Add a YieldTimer helper class for starvation avoidance.

view details

Graydon Hoare

commit sha 5f284f211995cee2b5338e0fcba529a9102329ea

Add Async{Read,Write} metrics to OverlayMetrics

view details

Graydon Hoare

commit sha 9697969a3e5ad0ac68aca6e442177c6a73f29476

Use a larger read buffer and synchronously read as much as possible.

view details

Graydon Hoare

commit sha 34281386f1fc3f52e859165dd98ffb8b8b5a3435

Switch from buffered to vectored async_writes.

view details

Graydon Hoare

commit sha e69c410ce42124ecc9bfcfe1101fa01708a2169f

Bounce off ASIO once when initiating messageSender.

view details

Graydon Hoare

commit sha 495216c9bb4782271d9d88e88a5b8f41f08bfbc4

Change straggler logic to use last-enqueued time, not last empty time.

view details

Graydon Hoare

commit sha 494655487960ffaa695b2d8c622b82f77e7596e5

TCPPeer: remove a bunch of extraneous uses of 'self'

view details

push time in 21 days

more