profile
viewpoint
Lucas Saldanha lucassaldanha @ConsenSys New Zealand http://www.lucassaldanha.com I'm a passionate software developer. Always trying to learn something new and looking for innovative ways of doing things that I already know.

lucassaldanha/eth-net-intelligence-api 3

Ethereum Network Intelligence API

lucassaldanha/besu-private-txs 1

Besu Private Transaction Example

lucassaldanha/besu-three-nodes-example 1

Docker compose config for a network with 3 Besu nodes with privacy enabled

lucassaldanha/java8-lamda-and-streams-presentation 1

A sample project with source examples used in the Java 8 Lamda & Streams presentation

ConsenSys/animals 0

A repo used on a Git workshop

iikirilov/besu 0

An enterprise-grade Java-based, Apache 2.0 licensed Ethereum client

lucassaldanha/animals 0

A repo used on a Git workshop

lucassaldanha/besu 0

An enterprise-grade Java-based, Apache 2.0 licensed Ethereum client

lucassaldanha/besu-docs 0

Documentation for Hyperledger Besu enterprise-grade Java-based, Apache 2.0 licensed Ethereum client

lucassaldanha/besu-sample-networks 0

Hyperledger Besu Ethereum client quick-start makes you able to simply test all Besu features.

pull request commenthyperledger/besu

Adds priv_getcode

Would you please add an entry on CHANGELOG.md under the 1.4.1 section mentioning that we are adding priv_getCode?

josh-richardson

comment created time in a day

Pull request review commenthyperledger/besu

Adds priv_getcode

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.api.jsonrpc.internal.privacy.methods.priv;++import static org.apache.logging.log4j.LogManager.getLogger;++import org.hyperledger.besu.ethereum.api.jsonrpc.RpcMethod;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.JsonRpcRequestContext;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.JsonRpcMethod;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.parameters.BlockParameter;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcResponse;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcSuccessResponse;+import org.hyperledger.besu.ethereum.api.query.BlockchainQueries;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.PrivacyParameters;+import org.hyperledger.besu.ethereum.privacy.PrivateStateRootResolver;++import java.util.Optional;++import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivGetCode implements JsonRpcMethod {++  private static final Logger LOG = getLogger();++  private final BlockchainQueries blockchain;+  private final PrivateStateRootResolver privateStateRootResolver;+  private final PrivacyParameters privacyParameters;++  public PrivGetCode(+      final BlockchainQueries blockchain,+      final PrivacyParameters privacyParameters,+      final PrivateStateRootResolver privateStateRootResolver) {+    this.privacyParameters = privacyParameters;+    this.blockchain = blockchain;+    this.privateStateRootResolver = privateStateRootResolver;+  }++  @Override+  public String getName() {+    return RpcMethod.PRIV_GET_CODE.getMethodName();+  }++  @Override+  public JsonRpcResponse response(final JsonRpcRequestContext requestContext) {+    LOG.trace("Executing {}", RpcMethod.PRIV_GET_CODE.getMethodName());++    final Address address =

We have standardized that for any priv_ methods that have an equivalent eth_ method, we will make the first parameter (index 0) the privacyGroupId. And all other parameters will be the same.

So, the expected params for priv_getCode are:

params: [
   'privacy_group_id',
   '0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b',
   '0x2'  // 2
]

Take a look at PrivCall implementation

josh-richardson

comment created time in a day

Pull request review commenthyperledger/besu

Adds priv_getcode

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.api.jsonrpc.internal.privacy.methods.priv;++import static org.apache.logging.log4j.LogManager.getLogger;++import org.hyperledger.besu.ethereum.api.jsonrpc.RpcMethod;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.JsonRpcRequestContext;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.JsonRpcMethod;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.parameters.BlockParameter;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcResponse;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcSuccessResponse;+import org.hyperledger.besu.ethereum.api.query.BlockchainQueries;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.PrivacyParameters;+import org.hyperledger.besu.ethereum.privacy.PrivateStateRootResolver;++import java.util.Optional;++import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivGetCode implements JsonRpcMethod {++  private static final Logger LOG = getLogger();++  private final BlockchainQueries blockchain;+  private final PrivateStateRootResolver privateStateRootResolver;+  private final PrivacyParameters privacyParameters;++  public PrivGetCode(+      final BlockchainQueries blockchain,+      final PrivacyParameters privacyParameters,+      final PrivateStateRootResolver privateStateRootResolver) {+    this.privacyParameters = privacyParameters;+    this.blockchain = blockchain;+    this.privateStateRootResolver = privateStateRootResolver;+  }++  @Override+  public String getName() {+    return RpcMethod.PRIV_GET_CODE.getMethodName();+  }++  @Override+  public JsonRpcResponse response(final JsonRpcRequestContext requestContext) {+    LOG.trace("Executing {}", RpcMethod.PRIV_GET_CODE.getMethodName());++    final Address address =+        Address.fromHexString(requestContext.getRequiredParameter(0, String.class));++    final BlockParameter blockParameter =+        requestContext.getRequiredParameter(1, BlockParameter.class);++    final String privacyGroupString = requestContext.getRequiredParameter(2, String.class);++    final Bytes32 privacyGroupId = Bytes32.wrap(Bytes.fromBase64String(privacyGroupString));++    final Hash latestStateRoot =+        privateStateRootResolver.resolveLastStateRoot(+            privacyGroupId,+            blockParameter.isNumeric() && blockParameter.getNumber().isPresent()

As I mentioned before, extending AbstractBlockParameterMethod should help you handling the block paramter.

josh-richardson

comment created time in a day

Pull request review commenthyperledger/besu

Adds priv_getcode

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.api.jsonrpc.internal.privacy.methods.priv;++import static org.apache.logging.log4j.LogManager.getLogger;++import org.hyperledger.besu.ethereum.api.jsonrpc.RpcMethod;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.JsonRpcRequestContext;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.JsonRpcMethod;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.parameters.BlockParameter;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcResponse;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcSuccessResponse;+import org.hyperledger.besu.ethereum.api.query.BlockchainQueries;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.PrivacyParameters;+import org.hyperledger.besu.ethereum.privacy.PrivateStateRootResolver;++import java.util.Optional;++import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivGetCode implements JsonRpcMethod {

We have an abstraction for methods that have the block paramater to handle named or numbers, etc. Take a look at PrivCall and how it extends AbstractBlockParameterMethod. I believe we should make PrivGetCode extend AbstractBlockParameterMethod.

josh-richardson

comment created time in a day

Pull request review commenthyperledger/besu

Adds priv_getcode

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.api.jsonrpc.internal.privacy.methods.priv;++import static org.apache.logging.log4j.LogManager.getLogger;++import org.hyperledger.besu.ethereum.api.jsonrpc.RpcMethod;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.JsonRpcRequestContext;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.JsonRpcMethod;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.parameters.BlockParameter;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcResponse;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcSuccessResponse;+import org.hyperledger.besu.ethereum.api.query.BlockchainQueries;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.PrivacyParameters;+import org.hyperledger.besu.ethereum.privacy.PrivateStateRootResolver;++import java.util.Optional;++import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivGetCode implements JsonRpcMethod {++  private static final Logger LOG = getLogger();++  private final BlockchainQueries blockchain;+  private final PrivateStateRootResolver privateStateRootResolver;+  private final PrivacyParameters privacyParameters;++  public PrivGetCode(+      final BlockchainQueries blockchain,+      final PrivacyParameters privacyParameters,

It looks like we are passing down the PrivacyParameter object only to get an instance of the privateStateArchive. I think we should change this to receive the privateStateArchive directly.

josh-richardson

comment created time in a day

Pull request review commenthyperledger/besu

Adds priv_getcode

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.api.jsonrpc.internal.privacy.methods.priv;++import static org.apache.logging.log4j.LogManager.getLogger;++import org.hyperledger.besu.ethereum.api.jsonrpc.RpcMethod;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.JsonRpcRequestContext;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.JsonRpcMethod;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.parameters.BlockParameter;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcResponse;+import org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcSuccessResponse;+import org.hyperledger.besu.ethereum.api.query.BlockchainQueries;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.PrivacyParameters;+import org.hyperledger.besu.ethereum.privacy.PrivateStateRootResolver;++import java.util.Optional;++import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivGetCode implements JsonRpcMethod {++  private static final Logger LOG = getLogger();++  private final BlockchainQueries blockchain;+  private final PrivateStateRootResolver privateStateRootResolver;+  private final PrivacyParameters privacyParameters;++  public PrivGetCode(+      final BlockchainQueries blockchain,+      final PrivacyParameters privacyParameters,+      final PrivateStateRootResolver privateStateRootResolver) {+    this.privacyParameters = privacyParameters;+    this.blockchain = blockchain;+    this.privateStateRootResolver = privateStateRootResolver;+  }++  @Override+  public String getName() {+    return RpcMethod.PRIV_GET_CODE.getMethodName();+  }++  @Override+  public JsonRpcResponse response(final JsonRpcRequestContext requestContext) {+    LOG.trace("Executing {}", RpcMethod.PRIV_GET_CODE.getMethodName());

We already have DEBUG log level for JSON-RPC calls. I don't think we need this one.

josh-richardson

comment created time in a day

push eventhyperledger/besu

pinges

commit sha 1b0dffcb3072f3eae583099535a3e7838c5707d9

Rename method (#412) * rename the method isPersistingState to isPersistingPrivateState because that is what it is used for Signed-off-by: Stefan Pingel <stefan.pingel@consensys.net> * rename the method isPersistingState to isPersistingPrivateState because that is what it is used for Signed-off-by: Stefan Pingel <stefan.pingel@consensys.net> * rename the method isPersistingState to isPersistingPrivateState because that is what it is used for Signed-off-by: Stefan Pingel <stefan.pingel@consensys.net>

view details

push time in 2 days

PR merged hyperledger/besu

Rename method

This is renaming the method isPersistingState() on the MessageFrame class to isPersistingPrivateState(), as this is what it is used for.

+22 -22

0 comment

4 changed files

pinges

pr closed time in 2 days

push eventlucassaldanha/web3js-eea

Lucas Saldanha

commit sha b7220ab940dd2759d6079c26424f375acdde003c

Adding MacOS common gitignore folders

view details

push time in 2 days

create barnchlucassaldanha/web3js-eea

branch : fix-ignore

created branch time in 2 days

Pull request review commentPegaSysEng/orion

Persistent nodes

 private void writePortsToFile(final Config config, final int nodePort, final int     };   } -  private KeyValueStore createStorage(final String storage, final Path storagePath) {-    String db = "routerdb";+  private KeyValueStore<Bytes, Bytes> createStorage(final String storage, final Path storagePath, String dbName) {

Sure thing! It doesn't need to be part of this.

atoulme

comment created time in 2 days

Pull request review commentPegaSysEng/orion

Persistent nodes

-/*

So this class isn't used anymore? It doesn't impact Orion's capabilities of using a relational db, does it? (just checking it wasn't a mistake) :)

atoulme

comment created time in 2 days

Pull request review commentPegaSysEng/orion

Persistent nodes

+/*+ * Copyright 2020 ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ */+package net.consensys.orion.network;++import static org.apache.tuweni.crypto.sodium.Box.KeyPair.random;+import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertFalse;+import static org.junit.jupiter.api.Assertions.assertTrue;++import net.consensys.orion.config.Config;++import java.net.URI;+import java.util.Collections;+import java.util.Iterator;+import java.util.Map;+import java.util.concurrent.ConcurrentHashMap;++import org.apache.tuweni.crypto.sodium.Box;+import org.apache.tuweni.kv.MapKeyValueStore;+import org.junit.jupiter.api.Test;++class PersistentNetworkNodesTest {

Could we add a test to ensure that we don't override a known node entry? This is to ensure we never allow discovery to override the table.

atoulme

comment created time in 2 days

Pull request review commentPegaSysEng/orion

Persistent nodes

 public Path workDir() {    * Storage engine used to save payloads and related information. Options:    *    * <ul>-   * <li>leveldb:path - LevelDB-   * <li>mapdb:path - MapDB+   * <li>leveldb:path - LevelDB</li>+   * <li>mapdb:path - MapDB</li>    * <li>sql:jdbcurl - Relational database</li>-   * <li>memory - Contents are cleared when Orion exits+   * <li>memory - Contents are cleared when Orion exits</li>    * </ul>    *    * <strong>Default:</strong> "leveldb"    *    * @return Storage string specifying a storage engine and/or storage path    */   public String storage() {-    return configuration.getString("storage");+    return System.getenv().getOrDefault("ORION_STORAGE", configuration.getString("storage"));+  }++  /**+   * Storage engine used to save node information. Options:+   *+   * <ul>+   * <li>leveldb:path - LevelDB</li>+   * <li>mapdb:path - MapDB</li>+   * <li>sql:jdbcurl - Relational database</li>+   * <li>memory - Contents are cleared when Orion exits</li>+   * </ul>+   *+   * <strong>Default:</strong> "memory"+   *+   * @return Storage string specifying a storage engine and/or storage path+   */+  public String nodeStorage() {+    return System.getenv().getOrDefault("ORION_NODE_STORAGE", configuration.getString("nodestorage"));

Once more, see my comment about this node storage name :)

atoulme

comment created time in 2 days

Pull request review commentPegaSysEng/orion

Persistent nodes

 private void writePortsToFile(final Config config, final int nodePort, final int     };   } -  private KeyValueStore createStorage(final String storage, final Path storagePath) {-    String db = "routerdb";+  private KeyValueStore<Bytes, Bytes> createStorage(final String storage, final Path storagePath, String dbName) {

I know that this isn't technically part of this PR. But it would be great if we could move this logic of creating the storage to its own class. Feel free to ignore this comment if you think it will take up too much of your time :)

atoulme

comment created time in 2 days

Pull request review commentPegaSysEng/orion

Persistent nodes

 private void writePortsToFile(final Config config, final int nodePort, final int     try (final FileOutputStream fileOutputStream = new FileOutputStream(portsFile)) {       properties.store(           fileOutputStream,-          "This file contains the ports used by the running instance of Pantheon. This file will be deleted after the node is shutdown.");+          "This file contains the ports used by the running instance of Besu. This file will be deleted after the node is shutdown.");

This was probably an error when copying this logic from Pantheon in the first place. I believe the ports file is used to write what ports the process (in this case Orion) is using. This is useful when we have dynamically allocated ports.

This message should probably read like:

This file contains the ports used by the running instance of Orion. This file will be deleted after the node is shutdown.
atoulme

comment created time in 2 days

Pull request review commentPegaSysEng/orion

Persistent nodes

   }    private final Vertx vertx;-  private KeyValueStore storage;+  private KeyValueStore<Bytes, Bytes> storage;+  private KeyValueStore<Bytes, Bytes> nodeStorage;

NIT: see my other comment about this name :)

atoulme

comment created time in 2 days

Pull request review commentPegaSysEng/orion

Persistent nodes

 public void stop() {         log.error("Error closing storage", e);       }     }+    if (nodeStorage != null) {

NIT: blank line before if statement

atoulme

comment created time in 2 days

Pull request review commentPegaSysEng/orion

Persistent nodes

 public static Config nodeConfig(       final String tls,       final String tlsServerTrust,       final String tlsClientTrust,-      final String storage) throws IOException {+      final String storage,+      final String nodeStorage) throws IOException {

NIT: nodeStorage might be a bit confusing. Given this is used to store the knownNodes information, maybe we can call this knownNodesStorage or knownNodesTableStorage?

Because we use the term node everywhere, when I readnodeStorage it kinda sounds as the storage of the node aka the "main" storage.

atoulme

comment created time in 2 days

Pull request review commentPegaSysEng/orion

Persistent nodes

 dependencyManagement {     dependency 'org.apache.tuweni:tuweni-crypto:0.10.0'     dependency 'org.apache.tuweni:tuweni-io:0.10.0'     dependency 'org.apache.tuweni:tuweni-junit:0.10.0'-    dependency 'org.apache.tuweni:tuweni-kv:0.10.0'+    dependency 'org.apache.tuweni:tuweni-kv:1.0.0-SNAPSHOT'

Do you think we'll be able to release Tuweni 1.0.0 before merging this into Orion?

atoulme

comment created time in 2 days

pull request commenthyperledger/besu

[MAINTAINERS] New maintainer candidate: Stefan Pingel

@pinges Please let us know if you accept this nomination!

lucassaldanha

comment created time in 4 days

PR opened hyperledger/besu

[MAINTAINERS] New maintainer candidate: Stefan Pingel

I propose to add Stefan as a Besu project maintainer.

Over the past few months, his contributions to the Private Transactions feature have added a lot of value to the project. You can see his past contributions cliking here. I have worked close to him and can attest that he is committed to the project and will help to keep this project in a high-quality standard.

Voting ends two weeks from today, midnight UTC on 2020-03-02 countdown

For more information on this process see the Becoming a Maintainer section in the MAINTAINERS.md file.

+1 -0

0 comment

1 changed file

pr created time in 4 days

create barnchlucassaldanha/besu

branch : new-maintainer-candidate

created branch time in 4 days

delete branch lucassaldanha/besu

delete branch : mainnet-privacy-2

delete time in 4 days

PR closed lucassaldanha/besu

Add participants to on chain privacy group

<!-- Thanks for sending a pull request! Please check out our contribution guidelines: --> <!-- https://github.com/hyperledger/besu/blob/master/CONTRIBUTING.md -->

PR description

Fixed Issue(s)

<!-- Please link to fixed issue(s) here using format: fixes #<issue number> --> <!-- Example: "fixes #2" -->

+3089 -79

0 comment

61 changed files

iikirilov

pr closed time in 4 days

PR closed lucassaldanha/besu

onchain privacy group creation

Signed-off-by: Ivaylo Kirilov iikirilov@gmail.com

<!-- Thanks for sending a pull request! Please check out our contribution guidelines: --> <!-- https://github.com/hyperledger/besu/blob/master/CONTRIBUTING.md -->

PR description

On-chain privacy group creation.

Includes privx_createPrivacyGroup. Modified priv_FindPrivacyGroup to append on chain privacy groups to result. Uses privx_FindPrivacyGroup to return on chain groups. priv_FindPrivacyGroup returms Orion managed privacy groups only. Uses (public) OnChainPrivacyPrecompiledContract to manage onchain privacy group interaction

Fixed Issue(s)

<!-- Please link to fixed issue(s) here using format: fixes #<issue number> --> <!-- Example: "fixes #2" -->

+1349 -58

0 comment

41 changed files

iikirilov

pr closed time in 4 days

delete branch lucassaldanha/besu

delete branch : mainnet-privacy

delete time in 4 days

delete branch lucassaldanha/besu

delete branch : updated-migration

delete time in 4 days

delete branch lucassaldanha/besu

delete branch : fix-private-tx-chainid

delete time in 4 days

delete branch lucassaldanha/besu

delete branch : private-state-migration

delete time in 4 days

push eventhyperledger/besu

Lucas Saldanha

commit sha 9fa9c858ead30d7d1cb82d6435cdaa1c815e5950

Private state update metadata and migration (#404) (backport from release-1.4) Private state update metadata and migration Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 4 days

PR merged hyperledger/besu

Private state update metadata and migration (#402)

(backport from release-1.4) Private state update metadata and migration

Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net

+3677 -446

0 comment

71 changed files

lucassaldanha

pr closed time in 4 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha 6164025aa86b6ad87a5d9a7b104f99c68ad8687d

Private state update metadata and migration (release 1.4) (#402) (backport from release-1.4) Private state update metadata and migration Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 4 days

PR closed hyperledger/besu

Reviewers
Private state update metadata and migration privacy

PR description

• Updated the private state storage with new data structures that aren't backwards compatible. • Added PrivateStateRootResolver that handles public chain forks and their impact on the private state • Created migration logic that will process previous private txs and re-create the data in the update 1.4.x format. • Migration is based on the schema version, defined in the PrivateStateStorage • Added flag that user needs to "opt-in" to migrate the data • Remove restriction for "latest" block from priv_call

+3637 -445

1 comment

68 changed files

lucassaldanha

pr closed time in 4 days

pull request commenthyperledger/besu

Private state update metadata and migration

Superseeded by https://github.com/hyperledger/besu/pull/404

lucassaldanha

comment created time in 4 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha d65bd5922cb16d6009093100f78f31d3fd35d472

Private state update metadata and migration (release 1.4) (#402) (backport from release-1.4) Private state update metadata and migration Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 4 days

PR opened hyperledger/besu

Private state update metadata and migration (#402)

(backport from release-1.4) Private state update metadata and migration

Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net

+3677 -445

0 comment

70 changed files

pr created time in 4 days

create barnchlucassaldanha/besu

branch : private-state-migration

created branch time in 4 days

delete branch lucassaldanha/besu

delete branch : updated-migration-1.4

delete time in 4 days

push eventhyperledger/besu

Lucas Saldanha

commit sha d7baae476509c2238c0ba113b63f89aa5dea1991

Private state update metadata and migration (release 1.4) (#402) Private state update metadata and migration Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 4 days

PR merged hyperledger/besu

Private state update metadata and migration (release 1.4) needed-rc privacy

PR description

• Updated the private state storage with new data structures that aren't backwards compatible. • Added PrivateStateRootResolver that handles public chain forks and their impact on the private state • Created migration logic that will process previous private txs and re-create the data in the update 1.4.x format. • Migration is based on the schema version, defined in the PrivateStateStorage • Added flag that user needs to "opt-in" to migrate the data • Remove restriction for "latest" block from priv_call

+3677 -445

0 comment

70 changed files

lucassaldanha

pr closed time in 4 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha 84174cf44d3f6ce298dea8c616b7835b2d1a9e9d

Renaming constants Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 7 days

Pull request review commenthyperledger/besu

Private state update metadata and migration (release 1.4)

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.privacy.storage.migration;++import static org.hyperledger.besu.ethereum.privacy.storage.PrivateStateKeyValueStorage.SCHEMA_VERSION_1_4_x;++import org.hyperledger.besu.ethereum.chain.Blockchain;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Block;+import org.hyperledger.besu.ethereum.core.BlockHeader;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.MutableWorldState;+import org.hyperledger.besu.ethereum.core.Transaction;+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;+import org.hyperledger.besu.ethereum.mainnet.ProtocolSpec;+import org.hyperledger.besu.ethereum.privacy.PrivateStateRootResolver;+import org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateStorage;+import org.hyperledger.besu.ethereum.privacy.storage.PrivacyGroupHeadBlockMap;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage;+import org.hyperledger.besu.ethereum.worldstate.WorldStateArchive;++import java.util.List;+import java.util.Optional;+import java.util.Set;+import java.util.function.Function;++import org.apache.logging.log4j.LogManager;+import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes32;++public class PrivateStorageMigration {++  private static final Logger LOG = LogManager.getLogger();++  private final Blockchain blockchain;+  private final Address privacyPrecompileAddress;+  private final ProtocolSchedule<?> protocolSchedule;+  private final WorldStateArchive publicWorldStateArchive;+  private final PrivateStateStorage privateStateStorage;+  private final PrivateStateRootResolver privateStateRootResolver;+  private final LegacyPrivateStateStorage legacyPrivateStateStorage;+  private final Function<ProtocolSpec<?>, PrivateMigrationBlockProcessor>+      privateMigrationBlockProcessorBuilder;++  public PrivateStorageMigration(+      final Blockchain blockchain,+      final Address privacyPrecompileAddress,+      final ProtocolSchedule<?> protocolSchedule,+      final WorldStateArchive publicWorldStateArchive,+      final PrivateStateStorage privateStateStorage,+      final PrivateStateRootResolver privateStateRootResolver,+      final LegacyPrivateStateStorage legacyPrivateStateStorage,+      final Function<ProtocolSpec<?>, PrivateMigrationBlockProcessor>+          privateMigrationBlockProcessorBuilder) {+    this.privateStateStorage = privateStateStorage;+    this.blockchain = blockchain;+    this.privacyPrecompileAddress = privacyPrecompileAddress;+    this.protocolSchedule = protocolSchedule;+    this.publicWorldStateArchive = publicWorldStateArchive;+    this.privateStateRootResolver = privateStateRootResolver;+    this.legacyPrivateStateStorage = legacyPrivateStateStorage;+    this.privateMigrationBlockProcessorBuilder = privateMigrationBlockProcessorBuilder;+  }++  public void migratePrivateStorage() {+    final long migrationStartTimestamp = System.currentTimeMillis();+    final long chainHeadBlockNumber = blockchain.getChainHeadBlockNumber();++    LOG.info("Migrating private storage database...");++    for (int blockNumber = 0; blockNumber <= chainHeadBlockNumber; blockNumber++) {+      final Block block =+          blockchain+              .getBlockByNumber(blockNumber)+              .orElseThrow(PrivateStorageMigrationException::new);+      final Hash blockHash = block.getHash();+      final BlockHeader blockHeader = block.getHeader();+      LOG.info("Processing block {} ({}/{})", blockHash, blockNumber, chainHeadBlockNumber);++      createPrivacyGroupHeadBlockMap(blockHeader);++      final int lastPmtIndex = findLastPMTIndexInBlock(block);+      if (lastPmtIndex >= 0) {+        final ProtocolSpec<?> protocolSpec = protocolSchedule.getByBlockNumber(blockNumber);+        final PrivateMigrationBlockProcessor privateMigrationBlockProcessor =+            privateMigrationBlockProcessorBuilder.apply(protocolSpec);++        final MutableWorldState publicWorldState =+            blockchain+                .getBlockHeader(blockHeader.getParentHash())+                .map(BlockHeader::getStateRoot)+                .flatMap(publicWorldStateArchive::getMutable)+                .orElseThrow(PrivateStorageMigrationException::new);++        final List<Transaction> transactionsToProcess =+            block.getBody().getTransactions().subList(0, lastPmtIndex + 1);+        final List<BlockHeader> ommers = block.getBody().getOmmers();++        privateMigrationBlockProcessor.processBlock(+            blockchain, publicWorldState, blockHeader, transactionsToProcess, ommers);+      }+    }++    if (isResultingPrivateStateRootAtHeadValid()) {+      privateStateStorage.updater().putDatabaseVersion(SCHEMA_VERSION_1_4_x).commit();

I'll rename the constants to SCHEMA_VERSION_1_0_0 and SCHEMA_VERSION_1_4_0 to make it clearer that they are specific to this release. Let's save the enum idea for when we need to change the version again! :)

lucassaldanha

comment created time in 7 days

Pull request review commenthyperledger/besu

Private state update metadata and migration (release 1.4)

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.privacy.storage.migration;++import static org.hyperledger.besu.ethereum.privacy.storage.PrivateStateKeyValueStorage.SCHEMA_VERSION_1_4_x;++import org.hyperledger.besu.ethereum.chain.Blockchain;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Block;+import org.hyperledger.besu.ethereum.core.BlockHeader;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.MutableWorldState;+import org.hyperledger.besu.ethereum.core.Transaction;+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;+import org.hyperledger.besu.ethereum.mainnet.ProtocolSpec;+import org.hyperledger.besu.ethereum.privacy.PrivateStateRootResolver;+import org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateStorage;+import org.hyperledger.besu.ethereum.privacy.storage.PrivacyGroupHeadBlockMap;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage;+import org.hyperledger.besu.ethereum.worldstate.WorldStateArchive;++import java.util.List;+import java.util.Optional;+import java.util.Set;+import java.util.function.Function;++import org.apache.logging.log4j.LogManager;+import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes32;++public class PrivateStorageMigration {++  private static final Logger LOG = LogManager.getLogger();++  private final Blockchain blockchain;+  private final Address privacyPrecompileAddress;+  private final ProtocolSchedule<?> protocolSchedule;+  private final WorldStateArchive publicWorldStateArchive;+  private final PrivateStateStorage privateStateStorage;+  private final PrivateStateRootResolver privateStateRootResolver;+  private final LegacyPrivateStateStorage legacyPrivateStateStorage;+  private final Function<ProtocolSpec<?>, PrivateMigrationBlockProcessor>+      privateMigrationBlockProcessorBuilder;++  public PrivateStorageMigration(+      final Blockchain blockchain,+      final Address privacyPrecompileAddress,+      final ProtocolSchedule<?> protocolSchedule,+      final WorldStateArchive publicWorldStateArchive,+      final PrivateStateStorage privateStateStorage,+      final PrivateStateRootResolver privateStateRootResolver,+      final LegacyPrivateStateStorage legacyPrivateStateStorage,+      final Function<ProtocolSpec<?>, PrivateMigrationBlockProcessor>+          privateMigrationBlockProcessorBuilder) {+    this.privateStateStorage = privateStateStorage;+    this.blockchain = blockchain;+    this.privacyPrecompileAddress = privacyPrecompileAddress;+    this.protocolSchedule = protocolSchedule;+    this.publicWorldStateArchive = publicWorldStateArchive;+    this.privateStateRootResolver = privateStateRootResolver;+    this.legacyPrivateStateStorage = legacyPrivateStateStorage;+    this.privateMigrationBlockProcessorBuilder = privateMigrationBlockProcessorBuilder;+  }++  public void migratePrivateStorage() {+    final long migrationStartTimestamp = System.currentTimeMillis();+    final long chainHeadBlockNumber = blockchain.getChainHeadBlockNumber();++    LOG.info("Migrating private storage database...");++    for (int blockNumber = 0; blockNumber <= chainHeadBlockNumber; blockNumber++) {+      final Block block =+          blockchain+              .getBlockByNumber(blockNumber)+              .orElseThrow(PrivateStorageMigrationException::new);+      final Hash blockHash = block.getHash();+      final BlockHeader blockHeader = block.getHeader();+      LOG.info("Processing block {} ({}/{})", blockHash, blockNumber, chainHeadBlockNumber);++      createPrivacyGroupHeadBlockMap(blockHeader);++      final int lastPmtIndex = findLastPMTIndexInBlock(block);+      if (lastPmtIndex >= 0) {+        final ProtocolSpec<?> protocolSpec = protocolSchedule.getByBlockNumber(blockNumber);+        final PrivateMigrationBlockProcessor privateMigrationBlockProcessor =+            privateMigrationBlockProcessorBuilder.apply(protocolSpec);++        final MutableWorldState publicWorldState =+            blockchain+                .getBlockHeader(blockHeader.getParentHash())+                .map(BlockHeader::getStateRoot)+                .flatMap(publicWorldStateArchive::getMutable)+                .orElseThrow(PrivateStorageMigrationException::new);++        final List<Transaction> transactionsToProcess =+            block.getBody().getTransactions().subList(0, lastPmtIndex + 1);

The sublist params are (inclusive, exclusive). We have a unit test checking the logic :)

lucassaldanha

comment created time in 7 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha 3808fe07507a5b9c3c496fccf67bff5439ae0ae5

PR comments Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 7 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu;++import static org.assertj.core.api.Assertions.assertThat;+import static org.hyperledger.besu.ethereum.privacy.PrivateStateRootResolver.EMPTY_ROOT_HASH;++import org.hyperledger.besu.config.GenesisConfigFile;+import org.hyperledger.besu.controller.BesuController;+import org.hyperledger.besu.controller.GasLimitCalculator;+import org.hyperledger.besu.crypto.SECP256K1;+import org.hyperledger.besu.enclave.Enclave;+import org.hyperledger.besu.enclave.EnclaveFactory;+import org.hyperledger.besu.enclave.types.SendResponse;+import org.hyperledger.besu.ethereum.ProtocolContext;+import org.hyperledger.besu.ethereum.chain.DefaultBlockchain;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Block;+import org.hyperledger.besu.ethereum.core.BlockDataGenerator;+import org.hyperledger.besu.ethereum.core.Difficulty;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.InMemoryStorageProvider;+import org.hyperledger.besu.ethereum.core.LogsBloomFilter;+import org.hyperledger.besu.ethereum.core.MiningParametersTestBuilder;+import org.hyperledger.besu.ethereum.core.PrivacyParameters;+import org.hyperledger.besu.ethereum.core.Transaction;+import org.hyperledger.besu.ethereum.core.Wei;+import org.hyperledger.besu.ethereum.eth.EthProtocolConfiguration;+import org.hyperledger.besu.ethereum.eth.sync.SynchronizerConfiguration;+import org.hyperledger.besu.ethereum.eth.transactions.TransactionPoolConfiguration;+import org.hyperledger.besu.ethereum.mainnet.HeaderValidationMode;+import org.hyperledger.besu.ethereum.privacy.PrivateStateRootResolver;+import org.hyperledger.besu.ethereum.privacy.PrivateTransaction;+import org.hyperledger.besu.ethereum.privacy.Restriction;+import org.hyperledger.besu.ethereum.privacy.storage.PrivacyGroupHeadBlockMap;+import org.hyperledger.besu.ethereum.privacy.storage.PrivacyStorageProvider;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage;+import org.hyperledger.besu.ethereum.privacy.storage.keyvalue.PrivacyKeyValueStorageProviderBuilder;+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPOutput;+import org.hyperledger.besu.ethereum.storage.keyvalue.KeyValueSegmentIdentifier;+import org.hyperledger.besu.metrics.noop.NoOpMetricsSystem;+import org.hyperledger.besu.plugin.services.storage.rocksdb.RocksDBKeyValuePrivacyStorageFactory;+import org.hyperledger.besu.plugin.services.storage.rocksdb.RocksDBKeyValueStorageFactory;+import org.hyperledger.besu.plugin.services.storage.rocksdb.RocksDBMetricsFactory;+import org.hyperledger.besu.plugin.services.storage.rocksdb.configuration.RocksDBFactoryConfiguration;+import org.hyperledger.besu.services.BesuConfigurationImpl;+import org.hyperledger.besu.testutil.TestClock;+import org.hyperledger.orion.testutil.OrionKeyConfiguration;+import org.hyperledger.orion.testutil.OrionTestHarness;+import org.hyperledger.orion.testutil.OrionTestHarnessFactory;++import java.io.IOException;+import java.math.BigInteger;+import java.net.URI;+import java.nio.file.Path;+import java.util.Arrays;+import java.util.Collections;+import java.util.List;+import java.util.stream.Collectors;++import io.vertx.core.Vertx;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;+import org.junit.Before;+import org.junit.Rule;+import org.junit.Test;+import org.junit.rules.TemporaryFolder;++@SuppressWarnings("rawtypes")+public class PrivacyReorgTest {+  private static final int MAX_OPEN_FILES = 1024;+  private static final long CACHE_CAPACITY = 8388608;+  private static final int MAX_BACKGROUND_COMPACTIONS = 4;+  private static final int BACKGROUND_THREAD_COUNT = 4;++  @Rule public final TemporaryFolder folder = new TemporaryFolder();++  private static final SECP256K1.KeyPair KEY_PAIR =+      SECP256K1.KeyPair.create(+          SECP256K1.PrivateKey.create(+              new BigInteger(+                  "8f2a55949038a9610f50fb23b5883af3b4ecb3c3bb792cbcefbd1542c692be63", 16)));+  private static final Bytes ENCLAVE_PUBLIC_KEY =+      Bytes.fromBase64String("A1aVtMxLCUHmBVHXoZzzBgPbW/wj5axDpW9X8l91SGo=");++  private static final String FIRST_BLOCK_WITH_NO_TRANSACTIONS_STATE_ROOT =+      "0x1bdf13f6d14c7322d6e695498aab258949e55574bef7eac366eb777f43d7dd2b";+  private static final String FIRST_BLOCK_WITH_SINGLE_TRANSACTION_STATE_ROOT =+      "0x16979b290f429e06d86a43584c7d8689d4292ade9a602e5c78e2867c6ebd904e";+  private static final String BLOCK_WITH_SINGLE_TRANSACTION_RECEIPTS_ROOT =+      "0xc8267b3f9ed36df3ff8adb51a6d030716f23eeb50270e7fce8d9822ffa7f0461";+  private static final String STATE_ROOT_AFTER_TRANSACTION_APPENDED_TO_EMTPY_STATE =+      "0x2121b68f1333e93bae8cd717a3ca68c9d7e7003f6b288c36dfc59b0f87be9590";+  private static final Bytes32 PRIVACY_GROUP_BYTES32 =+      Bytes32.fromHexString("0xf250d523ae9164722b06ca25cfa2a7f3c45df96b09e215236f886c876f715bfa");++  // EventEmitter contract binary+  private static final Bytes MOCK_PAYLOAD =+      Bytes.fromHexString(+          "0x608060405234801561001057600080fd5b5060008054600160a060020a03191633179055610199806100326000396000f3fe6080604052600436106100565763ffffffff7c01000000000000000000000000000000000000000000000000000000006000350416633fa4f245811461005b5780636057361d1461008257806367e404ce146100ae575b600080fd5b34801561006757600080fd5b506100706100ec565b60408051918252519081900360200190f35b34801561008e57600080fd5b506100ac600480360360208110156100a557600080fd5b50356100f2565b005b3480156100ba57600080fd5b506100c3610151565b6040805173ffffffffffffffffffffffffffffffffffffffff9092168252519081900360200190f35b60025490565b604080513381526020810183905281517fc9db20adedc6cf2b5d25252b101ab03e124902a73fcb12b753f3d1aaa2d8f9f5929181900390910190a16002556001805473ffffffffffffffffffffffffffffffffffffffff191633179055565b60015473ffffffffffffffffffffffffffffffffffffffff169056fea165627a7a72305820c7f729cb24e05c221f5aa913700793994656f233fe2ce3b9fd9a505ea17e8d8a0029");+  private static final PrivateTransaction PRIVATE_TRANSACTION =+      PrivateTransaction.builder()+          .chainId(BigInteger.valueOf(2018))+          .gasLimit(1000)+          .gasPrice(Wei.ZERO)+          .nonce(0)+          .payload(MOCK_PAYLOAD)+          .to(null)+          .privateFrom(ENCLAVE_PUBLIC_KEY)+          .privateFor(Collections.singletonList(ENCLAVE_PUBLIC_KEY))+          .restriction(Restriction.RESTRICTED)+          .value(Wei.ZERO)+          .signAndBuild(KEY_PAIR);++  private final BlockDataGenerator gen = new BlockDataGenerator();+  private BesuController besuController;+  private OrionTestHarness enclave;+  private PrivateStateRootResolver privateStateRootResolver;+  private PrivacyParameters privacyParameters;++  @Before+  public void setUp() throws IOException {+    // Start Enclave+    enclave =+        OrionTestHarnessFactory.create(+            folder.newFolder().toPath(),+            new OrionKeyConfiguration("enclavePublicKey", "enclavePrivateKey"));+    enclave.start();++    // Create Storage+    final Path dataDir = folder.newFolder().toPath();+    final Path dbDir = dataDir.resolve("database");++    // Configure Privacy+    privacyParameters =+        new PrivacyParameters.Builder()+            .setEnabled(true)+            .setStorageProvider(createKeyValueStorageProvider(dataDir, dbDir))+            .setEnclaveUrl(enclave.clientUrl())+            .setEnclaveFactory(new EnclaveFactory(Vertx.vertx()))+            .build();+    privacyParameters.setEnclavePublicKey("A1aVtMxLCUHmBVHXoZzzBgPbW/wj5axDpW9X8l91SGo=");

We can use ENCLAVE_PUBLIC_KEY.toBase64String()

lucassaldanha

comment created time in 7 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha dcd07e130b7406f8b8b2c0a96f25494c1709296b

Added private storage migration docs Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

Lucas Saldanha

commit sha 6b193c14221bd3067c3a16eb1e4b4cd140d02c5b

Updating changelog Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 7 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

           (PrivacyPrecompiledContract)               precompileContractRegistry.get(address, Account.DEFAULT_VERSION);       privacyPrecompiledContract.setPrivateTransactionProcessor(privateTransactionProcessor);+      if (AbstractBlockProcessor.class.isAssignableFrom(blockProcessor.getClass())) {

Maybe we can add something like:

if (AbstractBlockProcessor.class.isAssignableFrom(blockProcessor.getClass())) {
        blockProcessor =
            new PrivacyBlockProcessor(
                (AbstractBlockProcessor) blockProcessor,
                privacyParameters.getPrivateStateStorage());
      } else {
        throw new IllegalStateException("Unexpected error instantiating PrivacyBlockProcessor");
      }
lucassaldanha

comment created time in 7 days

PR opened hyperledger/besu

Private state update metadata and migration

PR description

• Updated the private state storage with new data structures that aren't backwards compatible. • Added PrivateStateRootResolver that handles public chain forks and their impact on the private state • Created migration logic that will process previous private txs and re-create the data in the update 1.4.x format. • Migration is based on the schema version, defined in the PrivateStateStorage • Added flag that user needs to "opt-in" to migrate the data • Remove restriction for "latest" block from priv_call

+3637 -445

0 comment

68 changed files

pr created time in 7 days

create barnchlucassaldanha/besu

branch : updated-migration-1.4

created branch time in 7 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha 08837d98d9d3c9d534b22a007106efafedfed309

Fix rawtypes warning Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 7 days

push eventlucassaldanha/besu

Danno Ferrin

commit sha 87f4829e235015ce166e2c0fbdb456ef40809d0d

More specific task metrics names (#389) A prior refactoring had accidentally removed the specific task names from the metrics labels. Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

view details

Danno Ferrin

commit sha ae3bd0129e94a96d1f569af7ee78f9925872ce0d

More trace fixes (#386) * pop flat trace context when handling halts * Better detection of precompiled and non-executed contracts * correct from address when calling in init code * fix some exotic nesting cases * correct from field for init code calls at depth >1 * correct cost on a non-call * changelog and notes Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

view details

Joshua Fernandes

commit sha 21e02b12b7e6efb4f676138d98fd1026d2d63727

adding the plugin-api javadoc jar at the root level (#378) Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>

view details

Antoine Toulme

commit sha 358458d5a74c16feb99f7e55159463d15f8122b6

BESU-56: remove erroneous links to errorprone checks (#385) Signed-off-by: Antoine Toulme <antoine@lunar-ocean.com>

view details

Jason Frame

commit sha 8b8a39c90dece4a9114a2fab8186e2c514d7205b

Changelog entry for multi-tenancy feature (#394) Signed-off-by: Edward Evans <edward.joshua.evans@gmail.com> Signed-off-by: Jason Frame <jasonwframe@gmail.com>

view details

Sally MacFarlane

commit sha 4c72ab7ef32abee63fc4b327a35368f4d8306a15

1.5 RC Changelog (#395) * 1.5 RC changelog additions Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>

view details

MadelineMurray

commit sha 80e0ee102ec1b95ff57353c301a437e994c0ac8c

Fixed link (#393) Signed-off-by: Madeline <madeline.murray@consensys.net> Co-authored-by: Edward <edjojob@gmail.com>

view details

Ratan Rai Sur

commit sha d06c8bfefc0ff037a8d6878296b9c3a17746c4f0

docker changelog (#391) * docker changelog Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com> * address comments Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com> Co-authored-by: MadelineMurray <43356962+MadelineMurray@users.noreply.github.com>

view details

MadelineMurray

commit sha 3728c3ced59172f58598b5b8b7c4bf6bccd5d4db

Added known bug to changelog (#388) * Added known bug Signed-off-by: Madeline <madeline.murray@consensys.net> * Added another known bug Signed-off-by: Madeline <madeline.murray@consensys.net>

view details

Joshua Fernandes

commit sha 9135622c128ba03a47b79fb9fc1a1b9cfdfb00e4

updating version to 1.4.0-rc1 (#397) Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>

view details

Joshua Fernandes

commit sha d82c8191d90fa234da8326cea6b150d4ccb696a5

updating verion to 1.4.1-snapshot (#398) Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>

view details

Abdelhamid Bakhta

commit sha 6677362598716d4a4519b9095bacb41a5e3c955b

[BESU-169] cache logs bloom filters automatically. (#367) * First iteration. Draft PR. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * fix SPDX header Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Use block broadcaster to index log bloom. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Remove useless toString method Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * spotless apply Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * cacheLogsBloomForBlockHeader Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * spotless apply Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * ensurePreviousSegmentsArePresent Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Added CLI flag to enable / disable automatic logs bloom indexing. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Create cache directory and cache file if not exist. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Fix acceptance test Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Write cache for block only if block is new canonical head. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Handling of chain reorg. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * fix Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * sportless apply Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Address PR comments. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Remove unused constant. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * spotless apply Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

view details

Danno Ferrin

commit sha bb0c9cb017cab05931f7975f4995b13e89622f66

[BESU-25] Use Devp2p Ping packets at v5 (#392) Broadcast that we support snappy compression Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

view details

Lucas Saldanha

commit sha 5c5b83cceebecc4cafe9fc7d034f862dfa84a0f2

Updated migration logic Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

Lucas Saldanha

commit sha f1ca3c52862765a91734b8dc7646dfaaa813bd6f

Merge branch 'master' into updated-migration Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 7 days

pull request commentethereum/remix-ide

[minor] Fixing typo on sample smart contract

Looks like the build is failing due to an error uploading the gist file that corresponds to the contract. I'm not sure if I'm doing something wrong or if it was an intermittent error.

lucassaldanha

comment created time in 8 days

push eventlucassaldanha/remix-ide

Lucas Saldanha

commit sha 0987e1f31cc8fe9342461edde59aad791914ab8a

[minor] Fixing typo on sample smart contract

view details

push time in 8 days

PR opened ethereum/remix-ide

[minor] Fixing typo on sample smart contract

Fixing typo on example-contracts.js contract s/retreive/retrieve

+2 -2

0 comment

1 changed file

pr created time in 8 days

create barnchlucassaldanha/remix-ide

branch : fix-typo

created branch time in 8 days

fork lucassaldanha/remix-ide

Browser-Only Ethereum IDE and Runtime Environment

https://remix-ide.readthedocs.io

fork in 8 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha ca1147614a32014dfff2b680e9bb9fe2692a3577

Fix AT Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

Lucas Saldanha

commit sha 177982f75365225f139aa75958d872d314c3a13e

[draft] private metadata migration v2 Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 8 days

startedamix/vimrc

started time in 9 days

push eventlucassaldanha/besu

Jason Frame

commit sha 9b35c3b65844580c1e897a9790d6d392d3945b3f

Validate private transaction before sending to enclave (#356) Signed-off-by: Jason Frame <jasonwframe@gmail.com>

view details

Danno Ferrin

commit sha 358ab092b46acc6469a9553e90f84946fefe876f

Trace API fixes (#377) - Correct Reporting of reverts in nested call - correct reporting and handling of value transfer in nested calls - correct handling of precompiles via DELEGATECALL & CALLCODE - Addition of precompiled contract gas costs - Re-work handling of storage writes - Initial handling of gas refunds - fix bug in DELEGATECALL tests, we don't need gas in the stack * this has a cascading effect on balances in diff tests - rework depth detection in flat trace - two new tests blocks Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

view details

Karim T

commit sha 39826b14231ef8cff7ca88589ed21972120b3e40

[BOUNTY-2] Add NAT Docker Support (#368) * add docker detection Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * add port mapping detection Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * add tests and refactor ip detection Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * clean RunnerBuilder Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * clean useless modification Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * spotless Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * resolve tests issues Signed-off-by: Karim TAAM <karim.t2am@gmail.com> * streamline auto detection Signed-off-by: Ratan Rai Sur <ratan.r.sur@gmail.com> Co-authored-by: Abdelhamid Bakhta <45264458+abdelhamidbakhta@users.noreply.github.com> Co-authored-by: Ratan Rai Sur <ratan.r.sur@gmail.com>

view details

mark-terry

commit sha d25297443e7327139acf9634ef4ab0a4544ee59e

[PIE-1798] Priv RPC acceptance tests with stub enclave. (#330) * [PIE-1798] Added some Privacy RPC ATs with a stub enclave. Signed-off-by: Mark Terry <mark.terry@consensys.net>

view details

Lucas Saldanha

commit sha 26539385684aa99037daedd86e1175a7dd5eebbe

Merge branch 'master' into updated-migration Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

Lucas Saldanha

commit sha e095b5be26a11a7724ae8b13b4b37f9adc7b58ad

Merge Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 10 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha 57194da630a4c26a3ffc26ea5d05b804ef7ed5dc

Handling enclave error Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 10 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha 9652a9d4f44fc902e053656bbb634f8c8c6c3491

Revert "Fix unused warnings" This reverts commit 118c7e5f Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

Lucas Saldanha

commit sha 16d938a8e6f853146b52a730d27b63c10903e475

Fix test Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 10 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha 118c7e5f16b2f5b9dd449277828417dcf066920b

Fix unused warnings Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 10 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha ea2af209972b6fc47351d86476bfb7b2a3199768

Minor PR comments Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

Lucas Saldanha

commit sha 6fce1b2df2120de7cc76fefbda72d9a4cc96a140

Reorg test Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

Lucas Saldanha

commit sha 65704c24b17e676f2b8c5a22e4f2ab7350c86c31

Spotless Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 10 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

 public void determineNonceForEeaRequestWithMoreThanOneMatchingGroupThrowsExcepti                     "privateFrom", new String[] {"first", "second"}, address, ENCLAVE_PUBLIC_KEY));   } -  @Test

Because all nonce logic now belongs in the NonceProvider tests. And we have the "equivalent" tests there.

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.privacy.storage.migration;++import org.hyperledger.besu.ethereum.chain.Blockchain;+import org.hyperledger.besu.ethereum.core.BlockHeader;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.MutableWorldState;+import org.hyperledger.besu.ethereum.debug.TraceOptions;+import org.hyperledger.besu.ethereum.mainnet.ProtocolSchedule;+import org.hyperledger.besu.ethereum.mainnet.ProtocolSpec;+import org.hyperledger.besu.ethereum.privacy.PrivateStateRootResolver;+import org.hyperledger.besu.ethereum.privacy.PrivateTransaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionProcessor;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionSimulatorResult;+import org.hyperledger.besu.ethereum.vm.BlockHashLookup;+import org.hyperledger.besu.ethereum.vm.DebugOperationTracer;+import org.hyperledger.besu.ethereum.worldstate.WorldStateArchive;++import java.util.Optional;++import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivateStorageMigrationTransactionProcessor {++  private final Blockchain blockchain;+  private final ProtocolSchedule<?> protocolSchedule;+  private final WorldStateArchive publicWorldStateArchive;+  private final WorldStateArchive privateWorldStateArchive;+  private final PrivateStateRootResolver privateStateRootResolver;++  public PrivateStorageMigrationTransactionProcessor(+      final Blockchain blockchain,+      final ProtocolSchedule<?> protocolSchedule,+      final WorldStateArchive publicWorldStateArchive,+      final WorldStateArchive privateWorldStateArchive,+      final PrivateStateRootResolver privateStateRootResolver) {+    this.blockchain = blockchain;+    this.protocolSchedule = protocolSchedule;+    this.publicWorldStateArchive = publicWorldStateArchive;+    this.privateWorldStateArchive = privateWorldStateArchive;+    this.privateStateRootResolver = privateStateRootResolver;+  }++  public Optional<PrivateTransactionSimulatorResult> process(+      final String privacyGroupId,+      final PrivateTransaction privateTransaction,+      final BlockHeader header) {+    if (header == null) {+      return Optional.empty();+    }++    final MutableWorldState publicWorldState =+        publicWorldStateArchive.getMutable(header.getStateRoot()).orElse(null);+    if (publicWorldState == null) {+      return Optional.empty();+    }++    // get the last world state root hash or create a new one+    final Bytes32 privacyGroupIdBytes = Bytes32.wrap(Bytes.fromBase64String(privacyGroupId));+    final Hash lastRootHash =+        privateStateRootResolver.resolveLastStateRoot(privacyGroupIdBytes, header.getHash());++    final MutableWorldState disposablePrivateState =+        privateWorldStateArchive+            .getMutable(lastRootHash)+            .orElseThrow(PrivateStorageMigrationException::new);++    final ProtocolSpec<?> protocolSpec = protocolSchedule.getByBlockNumber(header.getNumber());++    final PrivateTransactionProcessor privateTransactionProcessor =+        protocolSpec.getPrivateTransactionProcessor();++    final PrivateTransactionProcessor.Result result =+        privateTransactionProcessor.processTransaction(+            blockchain,+            publicWorldState.updater(),+            disposablePrivateState.updater(),+            header,+            privateTransaction,+            protocolSpec.getMiningBeneficiaryCalculator().calculateBeneficiary(header),+            new DebugOperationTracer(TraceOptions.DEFAULT),+            new BlockHashLookup(header, blockchain),+            privacyGroupIdBytes);++    final PrivateTransactionSimulatorResult txSimulatorResult =+        new PrivateTransactionSimulatorResult(+            privateTransaction, result, Optional.of(disposablePrivateState.rootHash()));++    if (result.isSuccessful()) {+      disposablePrivateState.persist();

All migration data is being created in the PrivateStorageMigration class.

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.privacy.storage.migration;++import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.EVENTS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.LOGS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.METADATA_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.OUTPUT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.REVERT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.STATUS_KEY_SUFFIX;++import org.hyperledger.besu.enclave.Enclave;+import org.hyperledger.besu.enclave.types.ReceiveResponse;+import org.hyperledger.besu.ethereum.chain.Blockchain;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Block;+import org.hyperledger.besu.ethereum.core.BlockHeader;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.Transaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionReceipt;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionSimulatorResult;+import org.hyperledger.besu.ethereum.privacy.storage.PrivacyGroupHeadBlockMap;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateBlockMetadata;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage.Updater;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateTransactionMetadata;+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;++import java.util.ArrayList;+import java.util.Base64;+import java.util.HashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.concurrent.atomic.AtomicLong;+import java.util.stream.Collectors;++import org.apache.logging.log4j.LogManager;+import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivateStorageMigration {++  private static final Logger LOG = LogManager.getLogger();++  private final PrivateStateStorage privateStateStorage;+  private final Blockchain blockchain;+  private final Enclave enclave;+  private final Bytes enclaveKey;+  private final Address privacyAddress;+  private final PrivateStorageMigrationTransactionProcessor transactionProcessor;++  private final List<String> migratedPrivacyGroups = new ArrayList<>();+  private final Map<Hash, List<Hash>> migratedTransactions = new HashMap<>();++  public PrivateStorageMigration(+      final PrivateStateStorage privateStateStorage,+      final Blockchain blockchain,+      final Enclave enclave,+      final Bytes enclaveKey,+      final Address privacyAddress,+      final PrivateStorageMigrationTransactionProcessor transactionProcessor) {+    this.privateStateStorage = privateStateStorage;+    this.blockchain = blockchain;+    this.enclave = enclave;+    this.enclaveKey = enclaveKey;+    this.privacyAddress = privacyAddress;+    this.transactionProcessor = transactionProcessor;+  }++  public void migratePrivateStorage() {+    long migrationStartTimestamp = System.currentTimeMillis();+    final AtomicLong numOfPrivateTxs = new AtomicLong();+    final long chainHeadBlockNumber = blockchain.getChainHeadBlockNumber();++    LOG.info("Migrating private storage database...");++    for (int blockNumber = 0; blockNumber <= chainHeadBlockNumber; blockNumber++) {+      final Block block =+          blockchain+              .getBlockByNumber(blockNumber)+              .orElseThrow(PrivateStorageMigrationException::new);+      final Hash blockHash = block.getHash();+      final BlockHeader blockHeader = block.getHeader();+      LOG.info("Processing block {} ({}/{})", blockHash, blockNumber, chainHeadBlockNumber);++      final PrivacyGroupHeadBlockMap privacyGroupHeadBlockMap =+          createPrivacyGroupHeadBlockMap(blockHeader);++      final List<Transaction> pmtsInBlock = findPMTsInBlock(block);+      for (int pmtIndex = 0; pmtIndex < pmtsInBlock.size(); pmtIndex++) {+        final Transaction pmt = pmtsInBlock.get(pmtIndex);+        LOG.trace("Processing PMT {} ({}/{})", pmt.getHash(), pmtIndex, pmtsInBlock.size() - 1);++        retrievePrivateTransactionFromEnclave(pmt)+            .ifPresent(+                receiveResponse -> {+                  final String privacyGroupId = receiveResponse.getPrivacyGroupId();+                  final PrivateTransaction privateTransaction =+                      parsePrivateTransaction(receiveResponse);+                  final Updater updater = privateStateStorage.updater();++                  final PrivateTransactionSimulatorResult result =+                      migratePrivateTransaction(+                          blockHeader, privacyGroupId, pmt, privateTransaction, updater);++                  if (result.isSuccessful()) {+                    updatePrivacyGroupHeadBlockMap(+                        blockHash, privacyGroupId, privacyGroupHeadBlockMap, updater);+                  }++                  migratedPrivacyGroups.add(privacyGroupId);++                  updater.commit();++                  numOfPrivateTxs.incrementAndGet();+                });+      }+    }++    privateStateStorage.updater().putDatabaseVersion(2).commit();++    deleteLegacyData();++    long migrationDuration = System.currentTimeMillis() - migrationStartTimestamp;+    LOG.info(+        "Migration took {} seconds to process {} blocks and migrate {} private transactions",+        migrationDuration / 1000.0,+        chainHeadBlockNumber,+        numOfPrivateTxs.get());+  }++  private Optional<ReceiveResponse> retrievePrivateTransactionFromEnclave(final Transaction pmt) {+    final ReceiveResponse receiveResponse;+    try {+      receiveResponse =+          enclave.receive(pmt.getPayload().toBase64String(), enclaveKey.toBase64String());+      return Optional.of(receiveResponse);+    } catch (Exception e) {+      return Optional.empty();+    }+  }++  private PrivacyGroupHeadBlockMap createPrivacyGroupHeadBlockMap(final BlockHeader blockHeader) {+    final PrivacyGroupHeadBlockMap privacyGroupHeadBlockHash =+        new PrivacyGroupHeadBlockMap(+            privateStateStorage+                .getPrivacyGroupHeadBlockMap(blockHeader.getParentHash())+                .orElse(PrivacyGroupHeadBlockMap.EMPTY));+    privateStateStorage+        .updater()+        .putPrivacyGroupHeadBlockMap(blockHeader.getHash(), privacyGroupHeadBlockHash)+        .commit();+    return privacyGroupHeadBlockHash;+  }++  private PrivateTransaction parsePrivateTransaction(final ReceiveResponse receiveResponse) {+    final BytesValueRLPInput bytesValueRLPInput =+        new BytesValueRLPInput(+            Bytes.wrap(Base64.getDecoder().decode(receiveResponse.getPayload())), false);+    final PrivateTransaction privateTransaction = PrivateTransaction.readFrom(bytesValueRLPInput);++    LOG.trace(+        "Retrieved private transaction {} for privacy group {}",+        privateTransaction.getHash(),+        receiveResponse.getPrivacyGroupId());++    return privateTransaction;+  }++  private PrivateTransactionSimulatorResult migratePrivateTransaction(+      final BlockHeader blockHeader,+      final String privacyGroupId,+      final Transaction privacyMarkerTransaction,+      final PrivateTransaction privateTransaction,+      final Updater updater) {+    final Hash blockHash = blockHeader.getHash();++    final PrivateTransactionSimulatorResult txResult =+        simulatePrivateTransaction(blockHeader, privacyGroupId, privateTransaction);++    if (txResult.isSuccessful()) {+      createPrivateBlockMetadata(+          blockHeader, privacyGroupId, privacyMarkerTransaction.getHash(), txResult, updater);++      createTransactionReceipt(blockHash, txResult, updater);++      if (!migratedTransactions.containsKey(blockHash)) {+        migratedTransactions.put(blockHash, new ArrayList<>());+      }+      migratedTransactions.get(blockHash).add(privateTransaction.getHash());+    }++    return txResult;+  }++  private PrivateTransactionSimulatorResult simulatePrivateTransaction(+      final BlockHeader blockHeader,

I have removed the method

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.privacy.storage.migration;++import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.EVENTS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.LOGS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.METADATA_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.OUTPUT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.REVERT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.STATUS_KEY_SUFFIX;++import org.hyperledger.besu.enclave.Enclave;+import org.hyperledger.besu.enclave.types.ReceiveResponse;+import org.hyperledger.besu.ethereum.chain.Blockchain;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Block;+import org.hyperledger.besu.ethereum.core.BlockHeader;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.Transaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionReceipt;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionSimulatorResult;+import org.hyperledger.besu.ethereum.privacy.storage.PrivacyGroupHeadBlockMap;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateBlockMetadata;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage.Updater;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateTransactionMetadata;+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;++import java.util.ArrayList;+import java.util.Base64;+import java.util.HashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.concurrent.atomic.AtomicLong;+import java.util.stream.Collectors;++import org.apache.logging.log4j.LogManager;+import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivateStorageMigration {++  private static final Logger LOG = LogManager.getLogger();++  private final PrivateStateStorage privateStateStorage;+  private final Blockchain blockchain;+  private final Enclave enclave;+  private final Bytes enclaveKey;+  private final Address privacyAddress;+  private final PrivateStorageMigrationTransactionProcessor transactionProcessor;++  private final List<String> migratedPrivacyGroups = new ArrayList<>();+  private final Map<Hash, List<Hash>> migratedTransactions = new HashMap<>();++  public PrivateStorageMigration(+      final PrivateStateStorage privateStateStorage,+      final Blockchain blockchain,+      final Enclave enclave,+      final Bytes enclaveKey,+      final Address privacyAddress,+      final PrivateStorageMigrationTransactionProcessor transactionProcessor) {+    this.privateStateStorage = privateStateStorage;+    this.blockchain = blockchain;+    this.enclave = enclave;+    this.enclaveKey = enclaveKey;+    this.privacyAddress = privacyAddress;+    this.transactionProcessor = transactionProcessor;+  }++  public void migratePrivateStorage() {+    long migrationStartTimestamp = System.currentTimeMillis();+    final AtomicLong numOfPrivateTxs = new AtomicLong();+    final long chainHeadBlockNumber = blockchain.getChainHeadBlockNumber();++    LOG.info("Migrating private storage database...");++    for (int blockNumber = 0; blockNumber <= chainHeadBlockNumber; blockNumber++) {+      final Block block =+          blockchain+              .getBlockByNumber(blockNumber)+              .orElseThrow(PrivateStorageMigrationException::new);+      final Hash blockHash = block.getHash();+      final BlockHeader blockHeader = block.getHeader();+      LOG.info("Processing block {} ({}/{})", blockHash, blockNumber, chainHeadBlockNumber);++      final PrivacyGroupHeadBlockMap privacyGroupHeadBlockMap =+          createPrivacyGroupHeadBlockMap(blockHeader);++      final List<Transaction> pmtsInBlock = findPMTsInBlock(block);+      for (int pmtIndex = 0; pmtIndex < pmtsInBlock.size(); pmtIndex++) {+        final Transaction pmt = pmtsInBlock.get(pmtIndex);+        LOG.trace("Processing PMT {} ({}/{})", pmt.getHash(), pmtIndex, pmtsInBlock.size() - 1);++        retrievePrivateTransactionFromEnclave(pmt)+            .ifPresent(+                receiveResponse -> {+                  final String privacyGroupId = receiveResponse.getPrivacyGroupId();+                  final PrivateTransaction privateTransaction =+                      parsePrivateTransaction(receiveResponse);+                  final Updater updater = privateStateStorage.updater();++                  final PrivateTransactionSimulatorResult result =+                      migratePrivateTransaction(+                          blockHeader, privacyGroupId, pmt, privateTransaction, updater);++                  if (result.isSuccessful()) {+                    updatePrivacyGroupHeadBlockMap(+                        blockHash, privacyGroupId, privacyGroupHeadBlockMap, updater);+                  }++                  migratedPrivacyGroups.add(privacyGroupId);++                  updater.commit();++                  numOfPrivateTxs.incrementAndGet();+                });+      }+    }++    privateStateStorage.updater().putDatabaseVersion(2).commit();++    deleteLegacyData();++    long migrationDuration = System.currentTimeMillis() - migrationStartTimestamp;+    LOG.info(+        "Migration took {} seconds to process {} blocks and migrate {} private transactions",+        migrationDuration / 1000.0,+        chainHeadBlockNumber,+        numOfPrivateTxs.get());+  }++  private Optional<ReceiveResponse> retrievePrivateTransactionFromEnclave(final Transaction pmt) {+    final ReceiveResponse receiveResponse;+    try {+      receiveResponse =+          enclave.receive(pmt.getPayload().toBase64String(), enclaveKey.toBase64String());+      return Optional.of(receiveResponse);+    } catch (Exception e) {+      return Optional.empty();+    }+  }++  private PrivacyGroupHeadBlockMap createPrivacyGroupHeadBlockMap(final BlockHeader blockHeader) {+    final PrivacyGroupHeadBlockMap privacyGroupHeadBlockHash =+        new PrivacyGroupHeadBlockMap(+            privateStateStorage+                .getPrivacyGroupHeadBlockMap(blockHeader.getParentHash())+                .orElse(PrivacyGroupHeadBlockMap.EMPTY));+    privateStateStorage+        .updater()+        .putPrivacyGroupHeadBlockMap(blockHeader.getHash(), privacyGroupHeadBlockHash)+        .commit();+    return privacyGroupHeadBlockHash;+  }++  private PrivateTransaction parsePrivateTransaction(final ReceiveResponse receiveResponse) {+    final BytesValueRLPInput bytesValueRLPInput =+        new BytesValueRLPInput(+            Bytes.wrap(Base64.getDecoder().decode(receiveResponse.getPayload())), false);+    final PrivateTransaction privateTransaction = PrivateTransaction.readFrom(bytesValueRLPInput);++    LOG.trace(+        "Retrieved private transaction {} for privacy group {}",+        privateTransaction.getHash(),+        receiveResponse.getPrivacyGroupId());++    return privateTransaction;+  }++  private PrivateTransactionSimulatorResult migratePrivateTransaction(+      final BlockHeader blockHeader,+      final String privacyGroupId,+      final Transaction privacyMarkerTransaction,+      final PrivateTransaction privateTransaction,+      final Updater updater) {+    final Hash blockHash = blockHeader.getHash();++    final PrivateTransactionSimulatorResult txResult =+        simulatePrivateTransaction(blockHeader, privacyGroupId, privateTransaction);++    if (txResult.isSuccessful()) {+      createPrivateBlockMetadata(+          blockHeader, privacyGroupId, privacyMarkerTransaction.getHash(), txResult, updater);++      createTransactionReceipt(blockHash, txResult, updater);

Same as before!

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.privacy.storage.migration;++import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.EVENTS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.LOGS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.METADATA_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.OUTPUT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.REVERT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.STATUS_KEY_SUFFIX;++import org.hyperledger.besu.enclave.Enclave;+import org.hyperledger.besu.enclave.types.ReceiveResponse;+import org.hyperledger.besu.ethereum.chain.Blockchain;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Block;+import org.hyperledger.besu.ethereum.core.BlockHeader;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.Transaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionReceipt;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionSimulatorResult;+import org.hyperledger.besu.ethereum.privacy.storage.PrivacyGroupHeadBlockMap;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateBlockMetadata;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage.Updater;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateTransactionMetadata;+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;++import java.util.ArrayList;+import java.util.Base64;+import java.util.HashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.concurrent.atomic.AtomicLong;+import java.util.stream.Collectors;++import org.apache.logging.log4j.LogManager;+import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivateStorageMigration {++  private static final Logger LOG = LogManager.getLogger();++  private final PrivateStateStorage privateStateStorage;+  private final Blockchain blockchain;+  private final Enclave enclave;+  private final Bytes enclaveKey;+  private final Address privacyAddress;+  private final PrivateStorageMigrationTransactionProcessor transactionProcessor;++  private final List<String> migratedPrivacyGroups = new ArrayList<>();+  private final Map<Hash, List<Hash>> migratedTransactions = new HashMap<>();++  public PrivateStorageMigration(+      final PrivateStateStorage privateStateStorage,+      final Blockchain blockchain,+      final Enclave enclave,+      final Bytes enclaveKey,+      final Address privacyAddress,+      final PrivateStorageMigrationTransactionProcessor transactionProcessor) {+    this.privateStateStorage = privateStateStorage;+    this.blockchain = blockchain;+    this.enclave = enclave;+    this.enclaveKey = enclaveKey;+    this.privacyAddress = privacyAddress;+    this.transactionProcessor = transactionProcessor;+  }++  public void migratePrivateStorage() {+    long migrationStartTimestamp = System.currentTimeMillis();+    final AtomicLong numOfPrivateTxs = new AtomicLong();+    final long chainHeadBlockNumber = blockchain.getChainHeadBlockNumber();++    LOG.info("Migrating private storage database...");++    for (int blockNumber = 0; blockNumber <= chainHeadBlockNumber; blockNumber++) {+      final Block block =+          blockchain+              .getBlockByNumber(blockNumber)+              .orElseThrow(PrivateStorageMigrationException::new);+      final Hash blockHash = block.getHash();+      final BlockHeader blockHeader = block.getHeader();+      LOG.info("Processing block {} ({}/{})", blockHash, blockNumber, chainHeadBlockNumber);++      final PrivacyGroupHeadBlockMap privacyGroupHeadBlockMap =+          createPrivacyGroupHeadBlockMap(blockHeader);++      final List<Transaction> pmtsInBlock = findPMTsInBlock(block);+      for (int pmtIndex = 0; pmtIndex < pmtsInBlock.size(); pmtIndex++) {+        final Transaction pmt = pmtsInBlock.get(pmtIndex);+        LOG.trace("Processing PMT {} ({}/{})", pmt.getHash(), pmtIndex, pmtsInBlock.size() - 1);++        retrievePrivateTransactionFromEnclave(pmt)+            .ifPresent(+                receiveResponse -> {+                  final String privacyGroupId = receiveResponse.getPrivacyGroupId();+                  final PrivateTransaction privateTransaction =+                      parsePrivateTransaction(receiveResponse);+                  final Updater updater = privateStateStorage.updater();++                  final PrivateTransactionSimulatorResult result =+                      migratePrivateTransaction(+                          blockHeader, privacyGroupId, pmt, privateTransaction, updater);++                  if (result.isSuccessful()) {+                    updatePrivacyGroupHeadBlockMap(+                        blockHash, privacyGroupId, privacyGroupHeadBlockMap, updater);+                  }++                  migratedPrivacyGroups.add(privacyGroupId);++                  updater.commit();++                  numOfPrivateTxs.incrementAndGet();+                });+      }+    }++    privateStateStorage.updater().putDatabaseVersion(2).commit();++    deleteLegacyData();++    long migrationDuration = System.currentTimeMillis() - migrationStartTimestamp;+    LOG.info(+        "Migration took {} seconds to process {} blocks and migrate {} private transactions",+        migrationDuration / 1000.0,+        chainHeadBlockNumber,+        numOfPrivateTxs.get());+  }++  private Optional<ReceiveResponse> retrievePrivateTransactionFromEnclave(final Transaction pmt) {+    final ReceiveResponse receiveResponse;+    try {+      receiveResponse =+          enclave.receive(pmt.getPayload().toBase64String(), enclaveKey.toBase64String());+      return Optional.of(receiveResponse);+    } catch (Exception e) {+      return Optional.empty();+    }+  }++  private PrivacyGroupHeadBlockMap createPrivacyGroupHeadBlockMap(final BlockHeader blockHeader) {+    final PrivacyGroupHeadBlockMap privacyGroupHeadBlockHash =+        new PrivacyGroupHeadBlockMap(+            privateStateStorage+                .getPrivacyGroupHeadBlockMap(blockHeader.getParentHash())+                .orElse(PrivacyGroupHeadBlockMap.EMPTY));+    privateStateStorage+        .updater()+        .putPrivacyGroupHeadBlockMap(blockHeader.getHash(), privacyGroupHeadBlockHash)+        .commit();+    return privacyGroupHeadBlockHash;+  }++  private PrivateTransaction parsePrivateTransaction(final ReceiveResponse receiveResponse) {+    final BytesValueRLPInput bytesValueRLPInput =+        new BytesValueRLPInput(+            Bytes.wrap(Base64.getDecoder().decode(receiveResponse.getPayload())), false);+    final PrivateTransaction privateTransaction = PrivateTransaction.readFrom(bytesValueRLPInput);++    LOG.trace(+        "Retrieved private transaction {} for privacy group {}",+        privateTransaction.getHash(),+        receiveResponse.getPrivacyGroupId());++    return privateTransaction;+  }++  private PrivateTransactionSimulatorResult migratePrivateTransaction(+      final BlockHeader blockHeader,+      final String privacyGroupId,+      final Transaction privacyMarkerTransaction,+      final PrivateTransaction privateTransaction,+      final Updater updater) {+    final Hash blockHash = blockHeader.getHash();++    final PrivateTransactionSimulatorResult txResult =+        simulatePrivateTransaction(blockHeader, privacyGroupId, privateTransaction);++    if (txResult.isSuccessful()) {+      createPrivateBlockMetadata(

Moving this into the transaction processor will hide this part of the migration logic inside it. The goal is to have all data structures created by the migration visible in the Migration class.

I'll rename it to make it clear that we are storing the data.

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.privacy.storage.migration;++import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.EVENTS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.LOGS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.METADATA_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.OUTPUT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.REVERT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.STATUS_KEY_SUFFIX;++import org.hyperledger.besu.enclave.Enclave;+import org.hyperledger.besu.enclave.types.ReceiveResponse;+import org.hyperledger.besu.ethereum.chain.Blockchain;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Block;+import org.hyperledger.besu.ethereum.core.BlockHeader;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.Transaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionReceipt;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionSimulatorResult;+import org.hyperledger.besu.ethereum.privacy.storage.PrivacyGroupHeadBlockMap;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateBlockMetadata;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage.Updater;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateTransactionMetadata;+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;++import java.util.ArrayList;+import java.util.Base64;+import java.util.HashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.concurrent.atomic.AtomicLong;+import java.util.stream.Collectors;++import org.apache.logging.log4j.LogManager;+import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivateStorageMigration {++  private static final Logger LOG = LogManager.getLogger();++  private final PrivateStateStorage privateStateStorage;+  private final Blockchain blockchain;+  private final Enclave enclave;+  private final Bytes enclaveKey;+  private final Address privacyAddress;+  private final PrivateStorageMigrationTransactionProcessor transactionProcessor;++  private final List<String> migratedPrivacyGroups = new ArrayList<>();+  private final Map<Hash, List<Hash>> migratedTransactions = new HashMap<>();++  public PrivateStorageMigration(+      final PrivateStateStorage privateStateStorage,+      final Blockchain blockchain,+      final Enclave enclave,+      final Bytes enclaveKey,+      final Address privacyAddress,+      final PrivateStorageMigrationTransactionProcessor transactionProcessor) {+    this.privateStateStorage = privateStateStorage;+    this.blockchain = blockchain;+    this.enclave = enclave;+    this.enclaveKey = enclaveKey;+    this.privacyAddress = privacyAddress;+    this.transactionProcessor = transactionProcessor;+  }++  public void migratePrivateStorage() {+    long migrationStartTimestamp = System.currentTimeMillis();+    final AtomicLong numOfPrivateTxs = new AtomicLong();+    final long chainHeadBlockNumber = blockchain.getChainHeadBlockNumber();++    LOG.info("Migrating private storage database...");++    for (int blockNumber = 0; blockNumber <= chainHeadBlockNumber; blockNumber++) {+      final Block block =+          blockchain+              .getBlockByNumber(blockNumber)+              .orElseThrow(PrivateStorageMigrationException::new);+      final Hash blockHash = block.getHash();+      final BlockHeader blockHeader = block.getHeader();+      LOG.info("Processing block {} ({}/{})", blockHash, blockNumber, chainHeadBlockNumber);++      final PrivacyGroupHeadBlockMap privacyGroupHeadBlockMap =+          createPrivacyGroupHeadBlockMap(blockHeader);++      final List<Transaction> pmtsInBlock = findPMTsInBlock(block);+      for (int pmtIndex = 0; pmtIndex < pmtsInBlock.size(); pmtIndex++) {+        final Transaction pmt = pmtsInBlock.get(pmtIndex);+        LOG.trace("Processing PMT {} ({}/{})", pmt.getHash(), pmtIndex, pmtsInBlock.size() - 1);++        retrievePrivateTransactionFromEnclave(pmt)+            .ifPresent(+                receiveResponse -> {+                  final String privacyGroupId = receiveResponse.getPrivacyGroupId();+                  final PrivateTransaction privateTransaction =+                      parsePrivateTransaction(receiveResponse);+                  final Updater updater = privateStateStorage.updater();++                  final PrivateTransactionSimulatorResult result =

Returning the result and checking if it was successful makes the code easier to read. I think we should keep it as it is.

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.privacy.storage.migration;++import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.EVENTS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.LOGS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.METADATA_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.OUTPUT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.REVERT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.STATUS_KEY_SUFFIX;++import org.hyperledger.besu.enclave.Enclave;+import org.hyperledger.besu.enclave.types.ReceiveResponse;+import org.hyperledger.besu.ethereum.chain.Blockchain;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Block;+import org.hyperledger.besu.ethereum.core.BlockHeader;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.Transaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionReceipt;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionSimulatorResult;+import org.hyperledger.besu.ethereum.privacy.storage.PrivacyGroupHeadBlockMap;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateBlockMetadata;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage.Updater;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateTransactionMetadata;+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;++import java.util.ArrayList;+import java.util.Base64;+import java.util.HashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.concurrent.atomic.AtomicLong;+import java.util.stream.Collectors;++import org.apache.logging.log4j.LogManager;+import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivateStorageMigration {++  private static final Logger LOG = LogManager.getLogger();++  private final PrivateStateStorage privateStateStorage;+  private final Blockchain blockchain;+  private final Enclave enclave;+  private final Bytes enclaveKey;+  private final Address privacyAddress;+  private final PrivateStorageMigrationTransactionProcessor transactionProcessor;++  private final List<String> migratedPrivacyGroups = new ArrayList<>();+  private final Map<Hash, List<Hash>> migratedTransactions = new HashMap<>();++  public PrivateStorageMigration(+      final PrivateStateStorage privateStateStorage,+      final Blockchain blockchain,+      final Enclave enclave,+      final Bytes enclaveKey,+      final Address privacyAddress,+      final PrivateStorageMigrationTransactionProcessor transactionProcessor) {+    this.privateStateStorage = privateStateStorage;+    this.blockchain = blockchain;+    this.enclave = enclave;+    this.enclaveKey = enclaveKey;+    this.privacyAddress = privacyAddress;+    this.transactionProcessor = transactionProcessor;+  }++  public void migratePrivateStorage() {+    long migrationStartTimestamp = System.currentTimeMillis();+    final AtomicLong numOfPrivateTxs = new AtomicLong();+    final long chainHeadBlockNumber = blockchain.getChainHeadBlockNumber();++    LOG.info("Migrating private storage database...");++    for (int blockNumber = 0; blockNumber <= chainHeadBlockNumber; blockNumber++) {+      final Block block =+          blockchain+              .getBlockByNumber(blockNumber)+              .orElseThrow(PrivateStorageMigrationException::new);+      final Hash blockHash = block.getHash();+      final BlockHeader blockHeader = block.getHeader();+      LOG.info("Processing block {} ({}/{})", blockHash, blockNumber, chainHeadBlockNumber);++      final PrivacyGroupHeadBlockMap privacyGroupHeadBlockMap =+          createPrivacyGroupHeadBlockMap(blockHeader);++      final List<Transaction> pmtsInBlock = findPMTsInBlock(block);+      for (int pmtIndex = 0; pmtIndex < pmtsInBlock.size(); pmtIndex++) {+        final Transaction pmt = pmtsInBlock.get(pmtIndex);+        LOG.trace("Processing PMT {} ({}/{})", pmt.getHash(), pmtIndex, pmtsInBlock.size() - 1);++        retrievePrivateTransactionFromEnclave(pmt)+            .ifPresent(+                receiveResponse -> {+                  final String privacyGroupId = receiveResponse.getPrivacyGroupId();+                  final PrivateTransaction privateTransaction =+                      parsePrivateTransaction(receiveResponse);+                  final Updater updater = privateStateStorage.updater();++                  final PrivateTransactionSimulatorResult result =+                      migratePrivateTransaction(+                          blockHeader, privacyGroupId, pmt, privateTransaction, updater);++                  if (result.isSuccessful()) {+                    updatePrivacyGroupHeadBlockMap(+                        blockHash, privacyGroupId, privacyGroupHeadBlockMap, updater);+                  }++                  migratedPrivacyGroups.add(privacyGroupId);++                  updater.commit();++                  numOfPrivateTxs.incrementAndGet();+                });+      }+    }++    privateStateStorage.updater().putDatabaseVersion(2).commit();++    deleteLegacyData();++    long migrationDuration = System.currentTimeMillis() - migrationStartTimestamp;+    LOG.info(+        "Migration took {} seconds to process {} blocks and migrate {} private transactions",+        migrationDuration / 1000.0,+        chainHeadBlockNumber,+        numOfPrivateTxs.get());+  }++  private Optional<ReceiveResponse> retrievePrivateTransactionFromEnclave(final Transaction pmt) {

Same as before :)

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.privacy.storage.migration;++import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.EVENTS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.LOGS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.METADATA_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.OUTPUT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.REVERT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.STATUS_KEY_SUFFIX;++import org.hyperledger.besu.enclave.Enclave;+import org.hyperledger.besu.enclave.types.ReceiveResponse;+import org.hyperledger.besu.ethereum.chain.Blockchain;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Block;+import org.hyperledger.besu.ethereum.core.BlockHeader;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.Transaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionReceipt;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionSimulatorResult;+import org.hyperledger.besu.ethereum.privacy.storage.PrivacyGroupHeadBlockMap;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateBlockMetadata;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage.Updater;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateTransactionMetadata;+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;++import java.util.ArrayList;+import java.util.Base64;+import java.util.HashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.concurrent.atomic.AtomicLong;+import java.util.stream.Collectors;++import org.apache.logging.log4j.LogManager;+import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivateStorageMigration {++  private static final Logger LOG = LogManager.getLogger();++  private final PrivateStateStorage privateStateStorage;+  private final Blockchain blockchain;+  private final Enclave enclave;+  private final Bytes enclaveKey;+  private final Address privacyAddress;+  private final PrivateStorageMigrationTransactionProcessor transactionProcessor;++  private final List<String> migratedPrivacyGroups = new ArrayList<>();+  private final Map<Hash, List<Hash>> migratedTransactions = new HashMap<>();++  public PrivateStorageMigration(+      final PrivateStateStorage privateStateStorage,+      final Blockchain blockchain,+      final Enclave enclave,+      final Bytes enclaveKey,+      final Address privacyAddress,+      final PrivateStorageMigrationTransactionProcessor transactionProcessor) {+    this.privateStateStorage = privateStateStorage;+    this.blockchain = blockchain;+    this.enclave = enclave;+    this.enclaveKey = enclaveKey;+    this.privacyAddress = privacyAddress;+    this.transactionProcessor = transactionProcessor;+  }++  public void migratePrivateStorage() {+    long migrationStartTimestamp = System.currentTimeMillis();+    final AtomicLong numOfPrivateTxs = new AtomicLong();+    final long chainHeadBlockNumber = blockchain.getChainHeadBlockNumber();++    LOG.info("Migrating private storage database...");++    for (int blockNumber = 0; blockNumber <= chainHeadBlockNumber; blockNumber++) {+      final Block block =+          blockchain+              .getBlockByNumber(blockNumber)+              .orElseThrow(PrivateStorageMigrationException::new);+      final Hash blockHash = block.getHash();+      final BlockHeader blockHeader = block.getHeader();+      LOG.info("Processing block {} ({}/{})", blockHash, blockNumber, chainHeadBlockNumber);++      final PrivacyGroupHeadBlockMap privacyGroupHeadBlockMap =+          createPrivacyGroupHeadBlockMap(blockHeader);++      final List<Transaction> pmtsInBlock = findPMTsInBlock(block);+      for (int pmtIndex = 0; pmtIndex < pmtsInBlock.size(); pmtIndex++) {+        final Transaction pmt = pmtsInBlock.get(pmtIndex);+        LOG.trace("Processing PMT {} ({}/{})", pmt.getHash(), pmtIndex, pmtsInBlock.size() - 1);++        retrievePrivateTransactionFromEnclave(pmt)+            .ifPresent(+                receiveResponse -> {+                  final String privacyGroupId = receiveResponse.getPrivacyGroupId();+                  final PrivateTransaction privateTransaction =+                      parsePrivateTransaction(receiveResponse);+                  final Updater updater = privateStateStorage.updater();++                  final PrivateTransactionSimulatorResult result =+                      migratePrivateTransaction(+                          blockHeader, privacyGroupId, pmt, privateTransaction, updater);++                  if (result.isSuccessful()) {+                    updatePrivacyGroupHeadBlockMap(

Same as before. Can we leave it as is?

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.privacy.storage.migration;++import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.EVENTS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.LOGS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.METADATA_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.OUTPUT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.REVERT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.STATUS_KEY_SUFFIX;++import org.hyperledger.besu.enclave.Enclave;+import org.hyperledger.besu.enclave.types.ReceiveResponse;+import org.hyperledger.besu.ethereum.chain.Blockchain;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Block;+import org.hyperledger.besu.ethereum.core.BlockHeader;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.Transaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionReceipt;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionSimulatorResult;+import org.hyperledger.besu.ethereum.privacy.storage.PrivacyGroupHeadBlockMap;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateBlockMetadata;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage.Updater;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateTransactionMetadata;+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;++import java.util.ArrayList;+import java.util.Base64;+import java.util.HashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.concurrent.atomic.AtomicLong;+import java.util.stream.Collectors;++import org.apache.logging.log4j.LogManager;+import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivateStorageMigration {++  private static final Logger LOG = LogManager.getLogger();++  private final PrivateStateStorage privateStateStorage;+  private final Blockchain blockchain;+  private final Enclave enclave;+  private final Bytes enclaveKey;+  private final Address privacyAddress;+  private final PrivateStorageMigrationTransactionProcessor transactionProcessor;++  private final List<String> migratedPrivacyGroups = new ArrayList<>();+  private final Map<Hash, List<Hash>> migratedTransactions = new HashMap<>();++  public PrivateStorageMigration(+      final PrivateStateStorage privateStateStorage,+      final Blockchain blockchain,+      final Enclave enclave,+      final Bytes enclaveKey,+      final Address privacyAddress,+      final PrivateStorageMigrationTransactionProcessor transactionProcessor) {+    this.privateStateStorage = privateStateStorage;+    this.blockchain = blockchain;+    this.enclave = enclave;+    this.enclaveKey = enclaveKey;+    this.privacyAddress = privacyAddress;+    this.transactionProcessor = transactionProcessor;+  }++  public void migratePrivateStorage() {+    long migrationStartTimestamp = System.currentTimeMillis();+    final AtomicLong numOfPrivateTxs = new AtomicLong();+    final long chainHeadBlockNumber = blockchain.getChainHeadBlockNumber();++    LOG.info("Migrating private storage database...");++    for (int blockNumber = 0; blockNumber <= chainHeadBlockNumber; blockNumber++) {+      final Block block =+          blockchain+              .getBlockByNumber(blockNumber)+              .orElseThrow(PrivateStorageMigrationException::new);+      final Hash blockHash = block.getHash();+      final BlockHeader blockHeader = block.getHeader();+      LOG.info("Processing block {} ({}/{})", blockHash, blockNumber, chainHeadBlockNumber);++      final PrivacyGroupHeadBlockMap privacyGroupHeadBlockMap =+          createPrivacyGroupHeadBlockMap(blockHeader);++      final List<Transaction> pmtsInBlock = findPMTsInBlock(block);+      for (int pmtIndex = 0; pmtIndex < pmtsInBlock.size(); pmtIndex++) {+        final Transaction pmt = pmtsInBlock.get(pmtIndex);+        LOG.trace("Processing PMT {} ({}/{})", pmt.getHash(), pmtIndex, pmtsInBlock.size() - 1);++        retrievePrivateTransactionFromEnclave(pmt)+            .ifPresent(+                receiveResponse -> {+                  final String privacyGroupId = receiveResponse.getPrivacyGroupId();+                  final PrivateTransaction privateTransaction =+                      parsePrivateTransaction(receiveResponse);

We could. But I'm trying to avoid changing the code not related to the migration (in case we need to revert it later).

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.privacy.storage.migration;++import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.EVENTS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.LOGS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.METADATA_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.OUTPUT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.REVERT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.STATUS_KEY_SUFFIX;++import org.hyperledger.besu.enclave.Enclave;+import org.hyperledger.besu.enclave.types.ReceiveResponse;+import org.hyperledger.besu.ethereum.chain.Blockchain;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Block;+import org.hyperledger.besu.ethereum.core.BlockHeader;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.Transaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionReceipt;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionSimulatorResult;+import org.hyperledger.besu.ethereum.privacy.storage.PrivacyGroupHeadBlockMap;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateBlockMetadata;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage.Updater;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateTransactionMetadata;+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;++import java.util.ArrayList;+import java.util.Base64;+import java.util.HashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.concurrent.atomic.AtomicLong;+import java.util.stream.Collectors;++import org.apache.logging.log4j.LogManager;+import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivateStorageMigration {++  private static final Logger LOG = LogManager.getLogger();++  private final PrivateStateStorage privateStateStorage;+  private final Blockchain blockchain;+  private final Enclave enclave;+  private final Bytes enclaveKey;+  private final Address privacyAddress;+  private final PrivateStorageMigrationTransactionProcessor transactionProcessor;++  private final List<String> migratedPrivacyGroups = new ArrayList<>();+  private final Map<Hash, List<Hash>> migratedTransactions = new HashMap<>();++  public PrivateStorageMigration(+      final PrivateStateStorage privateStateStorage,+      final Blockchain blockchain,+      final Enclave enclave,+      final Bytes enclaveKey,+      final Address privacyAddress,+      final PrivateStorageMigrationTransactionProcessor transactionProcessor) {+    this.privateStateStorage = privateStateStorage;+    this.blockchain = blockchain;+    this.enclave = enclave;+    this.enclaveKey = enclaveKey;+    this.privacyAddress = privacyAddress;+    this.transactionProcessor = transactionProcessor;+  }++  public void migratePrivateStorage() {+    long migrationStartTimestamp = System.currentTimeMillis();+    final AtomicLong numOfPrivateTxs = new AtomicLong();+    final long chainHeadBlockNumber = blockchain.getChainHeadBlockNumber();++    LOG.info("Migrating private storage database...");++    for (int blockNumber = 0; blockNumber <= chainHeadBlockNumber; blockNumber++) {+      final Block block =+          blockchain+              .getBlockByNumber(blockNumber)+              .orElseThrow(PrivateStorageMigrationException::new);+      final Hash blockHash = block.getHash();+      final BlockHeader blockHeader = block.getHeader();+      LOG.info("Processing block {} ({}/{})", blockHash, blockNumber, chainHeadBlockNumber);++      final PrivacyGroupHeadBlockMap privacyGroupHeadBlockMap =

I really don't wanna touch this migration logic unless we really need to. It looks like doing it the way we are doing isn't wrong, we just do more reads/writes.

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

+/*+ * Copyright ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.privacy.storage.migration;++import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.EVENTS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.LOGS_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.METADATA_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.OUTPUT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.REVERT_KEY_SUFFIX;+import static org.hyperledger.besu.ethereum.privacy.storage.LegacyPrivateStateKeyValueStorage.STATUS_KEY_SUFFIX;++import org.hyperledger.besu.enclave.Enclave;+import org.hyperledger.besu.enclave.types.ReceiveResponse;+import org.hyperledger.besu.ethereum.chain.Blockchain;+import org.hyperledger.besu.ethereum.core.Address;+import org.hyperledger.besu.ethereum.core.Block;+import org.hyperledger.besu.ethereum.core.BlockHeader;+import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.core.Transaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransaction;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionReceipt;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionSimulatorResult;+import org.hyperledger.besu.ethereum.privacy.storage.PrivacyGroupHeadBlockMap;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateBlockMetadata;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage.Updater;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateTransactionMetadata;+import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput;++import java.util.ArrayList;+import java.util.Base64;+import java.util.HashMap;+import java.util.List;+import java.util.Map;+import java.util.Optional;+import java.util.concurrent.atomic.AtomicLong;+import java.util.stream.Collectors;++import org.apache.logging.log4j.LogManager;+import org.apache.logging.log4j.Logger;+import org.apache.tuweni.bytes.Bytes;+import org.apache.tuweni.bytes.Bytes32;++public class PrivateStorageMigration {++  private static final Logger LOG = LogManager.getLogger();++  private final PrivateStateStorage privateStateStorage;+  private final Blockchain blockchain;+  private final Enclave enclave;+  private final Bytes enclaveKey;+  private final Address privacyAddress;+  private final PrivateStorageMigrationTransactionProcessor transactionProcessor;++  private final List<String> migratedPrivacyGroups = new ArrayList<>();+  private final Map<Hash, List<Hash>> migratedTransactions = new HashMap<>();++  public PrivateStorageMigration(+      final PrivateStateStorage privateStateStorage,+      final Blockchain blockchain,+      final Enclave enclave,+      final Bytes enclaveKey,+      final Address privacyAddress,+      final PrivateStorageMigrationTransactionProcessor transactionProcessor) {+    this.privateStateStorage = privateStateStorage;+    this.blockchain = blockchain;+    this.enclave = enclave;+    this.enclaveKey = enclaveKey;+    this.privacyAddress = privacyAddress;+    this.transactionProcessor = transactionProcessor;+  }++  public void migratePrivateStorage() {+    long migrationStartTimestamp = System.currentTimeMillis();+    final AtomicLong numOfPrivateTxs = new AtomicLong();

Nope, it is used within a lambda function so it needs to be final/effectively final.

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

 public PrivateStateKeyValueStorage(final KeyValueStorage keyValueStorage) {   }    @Override-  public Optional<Hash> getLatestStateRoot(final Bytes privacyId) {-    final byte[] id = privacyId.toArrayUnsafe();--    if (keyValueStorage.get(id).isPresent()) {-      return Optional.of(Hash.wrap(Bytes32.wrap(keyValueStorage.get(id).get())));-    } else {-      return Optional.empty();-    }-  }--  @Override-  public Optional<List<Log>> getTransactionLogs(final Bytes32 transactionHash) {-    final Optional<List<Log>> logs = get(transactionHash, LOGS_KEY_SUFFIX).map(this::rlpDecodeLog);-    if (logs.isEmpty()) {-      return get(transactionHash, EVENTS_KEY_SUFFIX).map(this::rlpDecodeLog);-    }-    return logs;-  }--  @Override-  public Optional<Bytes> getTransactionOutput(final Bytes32 transactionHash) {-    return get(transactionHash, OUTPUT_KEY_SUFFIX);+  public Optional<PrivateTransactionReceipt> getTransactionReceipt(+      final Bytes32 blockHash, final Bytes32 txHash) {+    final Bytes blockHashTxHash = Bytes.concatenate(blockHash, txHash);+    return get(blockHashTxHash, TX_RECEIPT_SUFFIX)+        .map(b -> PrivateTransactionReceipt.readFrom(new BytesValueRLPInput(b, false)));   }    @Override-  public Optional<Bytes> getStatus(final Bytes32 transactionHash) {-    return get(transactionHash, STATUS_KEY_SUFFIX);+  public Optional<PrivateBlockMetadata> getPrivateBlockMetadata(+      final Bytes32 blockHash, final Bytes32 privacyGroupId) {+    return get(Bytes.concatenate(blockHash, privacyGroupId), METADATA_KEY_SUFFIX)+        .map(this::rlpDecodePrivateBlockMetadata);   }    @Override-  public Optional<Bytes> getRevertReason(final Bytes32 transactionHash) {-    return get(transactionHash, REVERT_KEY_SUFFIX);+  public Optional<PrivacyGroupHeadBlockMap> getPrivacyGroupHeadBlockMap(final Bytes32 blockHash) {+    return get(blockHash, PRIVACY_GROUP_HEAD_BLOCK_MAP_SUFFIX)+        .map(b -> PrivacyGroupHeadBlockMap.readFrom(new BytesValueRLPInput(b, false)));   }    @Override-  public Optional<PrivateTransactionMetadata> getTransactionMetadata(-      final Bytes32 blockHash, final Bytes32 transactionHash) {-    return get(Bytes.concatenate(blockHash, transactionHash), METADATA_KEY_SUFFIX)-        .map(bytes -> PrivateTransactionMetadata.readFrom(new BytesValueRLPInput(bytes, false)));+  public int getSchemaVersion() {+    return get(Bytes.EMPTY, DB_VERSION_SUFFIX).map(Bytes::toInt).orElse(SCHEMA_VERSION_1_0_x);   }    @Override-  public boolean isPrivateStateAvailable(final Bytes32 transactionHash) {-    return false;+  public boolean isEmpty() {+    return keyValueStorage.getAllKeysThat(containsSuffix(LEGACY_STATUS_KEY_SUFFIX)).isEmpty()+        && keyValueStorage.getAllKeysThat(containsSuffix(TX_RECEIPT_SUFFIX)).isEmpty()+        && keyValueStorage.getAllKeysThat(containsSuffix(METADATA_KEY_SUFFIX)).isEmpty();   } -  @Override-  public boolean isWorldStateAvailable(final Bytes32 rootHash) {-    return false;+  private Predicate<byte[]> containsSuffix(final Bytes suffix) {+    return key ->+        key.length > suffix.toArrayUnsafe().length+            && Arrays.equals(+                Arrays.copyOfRange(key, key.length - suffix.toArrayUnsafe().length, key.length),+                suffix.toArrayUnsafe());   }    private Optional<Bytes> get(final Bytes key, final Bytes keySuffix) {     return keyValueStorage.get(Bytes.concatenate(key, keySuffix).toArrayUnsafe()).map(Bytes::wrap);   } -  private List<Log> rlpDecodeLog(final Bytes bytes) {-    return RLP.input(bytes).readList(Log::readFrom);+  private PrivateBlockMetadata rlpDecodePrivateBlockMetadata(final Bytes bytes) {

Isn't it inline already? :P

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

 public PrivateStateKeyValueStorage(final KeyValueStorage keyValueStorage) {   }    @Override-  public Optional<Hash> getLatestStateRoot(final Bytes privacyId) {-    final byte[] id = privacyId.toArrayUnsafe();--    if (keyValueStorage.get(id).isPresent()) {-      return Optional.of(Hash.wrap(Bytes32.wrap(keyValueStorage.get(id).get())));-    } else {-      return Optional.empty();-    }-  }--  @Override-  public Optional<List<Log>> getTransactionLogs(final Bytes32 transactionHash) {-    final Optional<List<Log>> logs = get(transactionHash, LOGS_KEY_SUFFIX).map(this::rlpDecodeLog);-    if (logs.isEmpty()) {-      return get(transactionHash, EVENTS_KEY_SUFFIX).map(this::rlpDecodeLog);-    }-    return logs;-  }--  @Override-  public Optional<Bytes> getTransactionOutput(final Bytes32 transactionHash) {-    return get(transactionHash, OUTPUT_KEY_SUFFIX);+  public Optional<PrivateTransactionReceipt> getTransactionReceipt(+      final Bytes32 blockHash, final Bytes32 txHash) {+    final Bytes blockHashTxHash = Bytes.concatenate(blockHash, txHash);+    return get(blockHashTxHash, TX_RECEIPT_SUFFIX)+        .map(b -> PrivateTransactionReceipt.readFrom(new BytesValueRLPInput(b, false)));   }    @Override-  public Optional<Bytes> getStatus(final Bytes32 transactionHash) {-    return get(transactionHash, STATUS_KEY_SUFFIX);+  public Optional<PrivateBlockMetadata> getPrivateBlockMetadata(+      final Bytes32 blockHash, final Bytes32 privacyGroupId) {+    return get(Bytes.concatenate(blockHash, privacyGroupId), METADATA_KEY_SUFFIX)+        .map(this::rlpDecodePrivateBlockMetadata);   }    @Override-  public Optional<Bytes> getRevertReason(final Bytes32 transactionHash) {-    return get(transactionHash, REVERT_KEY_SUFFIX);+  public Optional<PrivacyGroupHeadBlockMap> getPrivacyGroupHeadBlockMap(final Bytes32 blockHash) {+    return get(blockHash, PRIVACY_GROUP_HEAD_BLOCK_MAP_SUFFIX)+        .map(b -> PrivacyGroupHeadBlockMap.readFrom(new BytesValueRLPInput(b, false)));   }    @Override-  public Optional<PrivateTransactionMetadata> getTransactionMetadata(-      final Bytes32 blockHash, final Bytes32 transactionHash) {-    return get(Bytes.concatenate(blockHash, transactionHash), METADATA_KEY_SUFFIX)-        .map(bytes -> PrivateTransactionMetadata.readFrom(new BytesValueRLPInput(bytes, false)));+  public int getSchemaVersion() {+    return get(Bytes.EMPTY, DB_VERSION_SUFFIX).map(Bytes::toInt).orElse(SCHEMA_VERSION_1_0_x);

Renamed it to DB_VERSION_KEY

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

  import static java.nio.charset.StandardCharsets.UTF_8; -import org.hyperledger.besu.ethereum.core.Hash;-import org.hyperledger.besu.ethereum.core.Log;+import org.hyperledger.besu.ethereum.privacy.PrivateTransactionReceipt;+import org.hyperledger.besu.ethereum.privacy.storage.PrivateStateStorage.Updater; import org.hyperledger.besu.ethereum.rlp.BytesValueRLPInput; import org.hyperledger.besu.ethereum.rlp.RLP; import org.hyperledger.besu.plugin.services.storage.KeyValueStorage; import org.hyperledger.besu.plugin.services.storage.KeyValueStorageTransaction; -import java.util.List;+import java.util.Arrays; import java.util.Optional;+import java.util.function.Predicate;  import org.apache.tuweni.bytes.Bytes; import org.apache.tuweni.bytes.Bytes32;  public class PrivateStateKeyValueStorage implements PrivateStateStorage { -  @Deprecated private static final Bytes EVENTS_KEY_SUFFIX = Bytes.of("EVENTS".getBytes(UTF_8));+  public static final int SCHEMA_VERSION_1_0_x = 1;+  public static final int SCHEMA_VERSION_1_4_x = 2; -  private static final Bytes LOGS_KEY_SUFFIX = Bytes.of("LOGS".getBytes(UTF_8));-  private static final Bytes OUTPUT_KEY_SUFFIX = Bytes.of("OUTPUT".getBytes(UTF_8));+  private static final Bytes DB_VERSION_SUFFIX = Bytes.of("DBVERSION".getBytes(UTF_8));+  private static final Bytes TX_RECEIPT_SUFFIX = Bytes.of("RECEIPT".getBytes(UTF_8));   private static final Bytes METADATA_KEY_SUFFIX = Bytes.of("METADATA".getBytes(UTF_8));-  private static final Bytes STATUS_KEY_SUFFIX = Bytes.of("STATUS".getBytes(UTF_8));-  private static final Bytes REVERT_KEY_SUFFIX = Bytes.of("REVERT".getBytes(UTF_8));+  private static final Bytes PRIVACY_GROUP_HEAD_BLOCK_MAP_SUFFIX =+      Bytes.of("PGHEADMAP".getBytes(UTF_8));+  private static final Bytes LEGACY_STATUS_KEY_SUFFIX = Bytes.of("STATUS".getBytes(UTF_8));

I believe they are all being used :)

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

+/*+ * Copyright 2019 ConsenSys AG.+ *+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on+ * an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the+ * specific language governing permissions and limitations under the License.+ *+ * SPDX-License-Identifier: Apache-2.0+ */+package org.hyperledger.besu.ethereum.privacy.storage;++import org.hyperledger.besu.ethereum.core.Hash;+import org.hyperledger.besu.ethereum.rlp.RLPInput;+import org.hyperledger.besu.ethereum.rlp.RLPOutput;++import java.util.Collection;+import java.util.Collections;+import java.util.HashMap;+import java.util.List;+import java.util.Map;+import java.util.Objects;+import java.util.Set;++import org.apache.tuweni.bytes.Bytes32;++public class PrivacyGroupHeadBlockMap implements Map<Bytes32, Hash> {

I don't have a better idea. If we can't think of anything better I think we will just stick to this name.

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

+/*

I'll rename it to PrivateStorageMigrationTransactionProcessorResult. It is a bit too long but at least it won't confuse anyone :)

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

           (PrivacyPrecompiledContract)               precompileContractRegistry.get(address, Account.DEFAULT_VERSION);       privacyPrecompiledContract.setPrivateTransactionProcessor(privateTransactionProcessor);+      if (AbstractBlockProcessor.class.isAssignableFrom(blockProcessor.getClass())) {

When we are using privacy. We "wrap" the block processor (either the "mainnet" one of the "ethereum classic") with our privacy block processor.

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

 public PrivacyParameters getPrivacyParameters() {     return privacyParameters;   } +  public PrivateNonceProvider getPrivateNonceProvider() {

I believe we don't need getPrivateNonceProvider() anymore. We definitely need PrivateNonceProvider though. :) I have removed the method.

lucassaldanha

comment created time in 11 days

Pull request review commenthyperledger/besu

Private state update metadata and migration

 private PrivacyParameters privacyParameters() {       }     } -    return privacyParametersBuilder.build();+    final PrivacyParameters privacyParameters = privacyParametersBuilder.build();++    if (isPrivacyEnabled) {+      preSynchronizationTaskRunner.addTask(

Because part of the task is to evaluate the different triggering conditions. And one of the conditions is: you have a private db that needs migration but you don't have the migratePrivateDatabase flag enabled.

lucassaldanha

comment created time in 11 days

PR closed lucassaldanha/besu

Updated migration
+20878 -4652

0 comment

336 changed files

lucassaldanha

pr closed time in 11 days

push eventlucassaldanha/besu

Danno Ferrin

commit sha f4b921b7fc84759dfc62cb01193b7d917ca94ce0

Plugin error stack traces (#369) Because of how the Log4J2 api works exception stack traces were not being printed. Update to use the explicit "throwable" overloaded methods. Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

view details

Danno Ferrin

commit sha 059f0ae5f525b9ccb36bf9cc6ea997765759d307

VM Trace fixes (#372) * correct refund addresses * correct returned memory from static precompiled calls. * update integration test * precompiles sometimes get plain old CALLs Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

view details

Lucas Saldanha

commit sha f4da438a3fb9bdfb8600f5a888e91dac51dde311

Merge branch 'master' into updated-migration Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 11 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha aa0f1a6d55d0332ed7dbd9f38132cc7448ac9836

Changed migration msg Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 16 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha 8ea2920f5df44930de8f732ca5dcb1f61a668993

Changed migration msg

view details

push time in 16 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha 5111882b16196d93e9a647bb15062120d143e93f

Fix unit test Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 16 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha 39f920f406a6043c60c33e4f8ebdce9111d6dfcc

Fix unit test

view details

push time in 16 days

push eventlucassaldanha/besu

Abdelhamid Bakhta

commit sha 21dc78af61cdc1b3ffef606426701c0318697991

Renamed .index to .cache. (#364) Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

view details

Danno Ferrin

commit sha a933179254c8513b1c3722a077650a91ddd854d5

Json Blocks Import timestamp (#358) Allows for the setting of the timestamp in JSON block importer. Ordinarily it is now and then 1 second for each block. The initial time can now be set via CLI option `--start-time`, which defaults to now. Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

view details

Lucas Saldanha

commit sha 73adcf1fd55d42b8af9b9f78e70f15e106e91023

Fix import Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

Danno Ferrin

commit sha a7d3896af822c8028b718396971677088cc4f22e

More tracing api fixes (#361) * Handle CREATE and CREATE2 in flat trace * precompile calls don't generate traces * Handle REVERT in output and result nodes. * output doesn't need to be held behind a supplier * Handle padded output values when end is past memory * Store memory in vm trace for MLOADs * CREATE and CREATE2 store memory in vm-trace Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

view details

Lucas Saldanha

commit sha 174021110de164438c39063fcfb6c3bfe11f66ef

Merge branch 'master' into updated-migration Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 16 days

push eventlucassaldanha/besu

MadelineMurray

commit sha 8577e667bda21e268ef5862f574f6341e0c6dd6f

Adding Danno's changes back (#355) Signed-off-by: Madeline <madeline.murray@consensys.net>

view details

Sally MacFarlane

commit sha 13b1ca6baee0657ffa3992c8794bad1abe60640b

[MINOR] typo (#357) * typo Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>

view details

Paul Harris

commit sha 20e9cbbc7ead085c13f1101c6a6a29fa5a7bcc0c

[BESU-184] Update PicoCLI to latest version (#349) - Updated PicoCLI from version 3.9.6 to 4.1.4 and fixed test cases that were impacted by breaking changes of PicoCLI. - Updated the command factory to fallback to the default factory, as directed in the PicoCli v.4.0.0 release notes. Signed-off-by: Paul Harris <paul.harris@consensys.net>

view details

Danno Ferrin

commit sha 7a7416f74e8b509eb39c47ad6f1423563c64a00d

Handle CREATE and CREATE2 in flat trace (#353) * Handle CREATE and CREATE2 in flat trace Handle create and create2 operations. Some other tests also got adjusted as the source accounts were running out of test ether. Bumping up the value in the genesis caused the statediff values to also go up. Signed-off-by: Danno Ferrin <danno.ferrin@gmail.com>

view details

Abdelhamid Bakhta

commit sha d7d82e417d78e93b6db1386e87d72455860ee45a

[BESU-176] JRPC response to eth_getTransactionCount is different from other node implementations. (#351) * Fix strategy for getting next nonce for sender in `PendingTransactions`. Added test. Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * Introduced `TransactionsForSenderInfo` structure to have a less naive implementation, more robust against load testing. A queue of gaps between used nonces is maintained. `getNextNonceForSender` has been updated and have the following logic: - if no infos corresponding to the nonce returns an empty optional - if gap list is not empty returns the lowest nonce in the gap - else returns the max nonce + 1 Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * spotless Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * spotless Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * added test Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net> * spotless Signed-off-by: Abdelhamid Bakhta <abdelhamid.bakhta@consensys.net>

view details

Joshua Fernandes

commit sha f12c92ec58a95a485c1b9bdcf9a1257f3cd3024a

add the binary on ci at build time, dont need it to be in the code base (#359) Signed-off-by: Joshua Fernandes <joshua.fernandes@consensys.net>

view details

Sally MacFarlane

commit sha 9dfed5ab74cd84ac666b1367d08cab84af80e29b

[BESU-80] TLS to orion (#324) * enable SSL on connection to enclave Signed-off-by: Sally MacFarlane <sally.macfarlane@consensys.net>

view details

Lucas Saldanha

commit sha 2d7ea371fa5573d78b2b11288c943b7abfb7a38e

Remove restriction for priv_call Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

Lucas Saldanha

commit sha b6d28554ec4b045431de404d5a2a84ccf0f12bfa

Merge branch 'master' into updated-migration Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 17 days

PR closed hyperledger/besu

Reviewers
Fix private transaction chainId validation privacy

PR description

• Fixes the scenario when a private transaction doesn't have a chainId and the validator expects a chainId • Added wrong chain id error msg in JSON-RPC response

+20 -10

0 comment

4 changed files

lucassaldanha

pr closed time in 17 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha 88c4f93c296bfe6a1d500649b006ffeff04f276b

Fix private transaction chainId validation Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 17 days

PR opened hyperledger/besu

Fix private transaction chainId validation

PR description

• Fixes the scenario when a private transaction doesn't have a chainId and the validator expects a chainId • Added wrong chain id error msg in JSON-RPC response

+24 -11

0 comment

4 changed files

pr created time in 17 days

create barnchlucassaldanha/besu

branch : fix-private-tx-chainid

created branch time in 17 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha c28fb51b2ec52bed4e940b649fb2bb0382b267bb

Fix migration of failed txs Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 18 days

push eventlucassaldanha/besu

Lucas Saldanha

commit sha 8d9f4aadc8936f1c808c78a27aee96053b529d6a

Private state update metadata and migration Signed-off-by: Lucas Saldanha <lucas.saldanha@consensys.net>

view details

push time in 19 days

PR opened hyperledger/besu

Reviewers
Private state update metadata and migration

PR description

• Updated the private state storage with new data structures that aren't backwards compatible. • Added PrivateStateRootResolver that handles public chain forks and their impact on the private state • Created migration logic that will process previous private txs and re-create the data in the update 1.4.x format. • Added flag that user needs to "opt-in" to migrate the data

+3741 -415

0 comment

61 changed files

pr created time in 19 days

more