profile
viewpoint

DocNow/unshrtn 27

A LevelDB backed URL unshortening microservice written in JavaScript

acdha/NativeImaging 25

PIL-compatible interface for platform libraries such as GraphicsMagick, Aware or JAI.

eikeon/funnelsort 8

Lazy funnel sort implementation in golang

eikeon/dynamodb 3

A complete client side implementation of the DynamoDB API Version 2012-08-10 along with methods for mapping between items and go values (structs).

edsu/mediator 2

Look at Medium through Twitter

eikeon/aws4 1

A Go package for AWS Signature version 4

eikeon/brikker 1

A server for generating and serving image tiles from a jp2 source image(s).

eikeon/django-memorious 1

django application for dealing with static resources regarding urls, cache headers, and versions.

eikeon/ace 0

Ajax.org Cloud9 Editor

eikeon/alexa-nodekit 0

Amazon Echo Alexa's App Kit NodeJS Implementation

pull request commentaws/aws-cdk

feat(stepfunctions): new service integration classes that merge that task and state functionality

Looking good and looking forward to trying these out

shivlaks

comment created time in 5 days

issue openedaws/aws-sdk-go-v2

s3's ListPartsOutput's PartNumberMarker not as documented

Confirm by changing [ ] to [x] below to ensure that it's a bug:

Describe the bug

s3's ListPartsOutput's PartNumberMarker field is documented the same as its NextPartNumberMarker field. But does not return the same ;) NextPartNumberMarker works as documented and PartNumberMarker does not.

See: https://github.com/aws/aws-sdk-go-v2/blob/40a22fcb842831dd1511cc4a79f9fdc3c308e407/service/s3/api_op_ListParts.go#L201

Version of AWS SDK for Go?

v0.21.0

Version of Go (go version)?

v1.14

To Reproduce (observed behavior) Steps to reproduce the behavior (please share code or minimal repo)

make a ListPartsRequest

Expected behavior

expect a NextPartNumberMarker

Additional context

Instead get PartNumberMarker of 0 over and over :(

created time in 6 days

issue commentaws/aws-cdk

tasks.DynamoAttributeValue().withB double encoding value?

@wong-a Yes, I was expecting the binary value to behave like the low-level API. And since we’re in this JSON-based context passing binary with type string — I assumed it’d need to be expecting base64 encoded string for it to work.

I’ve been passing the binary around with the various AWS SDKs as you describe. And by the time the data gets to were I was trying to use tasks.DynamoUpdateItem above within the state machine the ContentID is already base64 encoded since ASL is JSON-based. Normally if the SDKs encode something on the way in they decode it on the way out or the other way around.

eikeon

comment created time in 19 days

issue commentaws/aws-cdk

tasks.DynamoAttributeValue().withB double encoding value?

Just reporting back that using a lambda for this task instead of the DynamoUpdateItem step functions service integration worked to avoid this issue. And I can have the lambda grab the current time instead of relying on a separate EvaluateExpression task.

eikeon

comment created time in 19 days

issue commentaws/aws-cdk

tasks.DynamoAttributeValue().withB double encoding value?

@shivlaks Thank you; In the meantime I'm switching to a Lambda for this bit. But can help retest or keep digging if need be.

eikeon

comment created time in 21 days

issue commentaws/aws-cdk

tasks.DynamoAttributeValue().withB double encoding value?

CDK seems to be doing its part. It's generating the following bit in the state machine definition:

        "Record FetchCompleted": {
            "Next": "Publish FetchCompleted",
            "Parameters": {
                "Key": { "ContentID": { "B.$": "$.ContentID" } },
                "TableName": "content-LabelsBA41E4AA-1XBL4K17A6P60",
                "ExpressionAttributeValues": {
                    ":now": { "N.$": "$.FetchCompletedTime" }
                },
                "UpdateExpression": "SET FetchCompletedTime = :now REMOVE FetchStartedTime"
            },
            "Type": "Task",
            "Resource": "arn:aws:states:::dynamodb:updateItem",
            "ResultPath": "$.RecordFetchCompleted"
        },

Is there somewhere to file a bug against the step function integration services? Where is should be interpreting that "$.ContentID" as a base64 encoded string and converting it to a blob that's expected there

eikeon

comment created time in 23 days

issue commentaws/aws-cdk

RunLambdaTask with outputPath not working

Doh, linked the above merge request accidentally to this one in a commit even though the title had the correct issue #

eikeon

comment created time in 23 days

PR closed aws/aws-cdk

fix(stepfunctions): AttributeValue.withB

Commit Message

fix(stepfunctions): AttributeValue.withB

Fix type with conversion from base64

Close #7709

End Commit Message


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

+0 -0

5 comments

0 changed file

eikeon

pr closed time in 23 days

push eventeikeon/aws-cdk

push time in 23 days

push eventeikeon/aws-cdk

Daniel Krech

commit sha e55421464e2cb6c029f8cae8cdd066ba20deda82

WIP: Close #7724

view details

push time in 23 days

Pull request review commentaws/aws-cdk

fix(stepfunctions): AttributeValue.withB

 export class DynamoAttributeValue {    * Sets an attribute of type Binary. For example:  "B": "dGhpcyB0ZXh0IGlzIGJhc2U2NC1lbmNvZGVk"    */   public withB(value: string) {-    this.attributeValue.B = value;+    this.attributeValue.B = Buffer.from(value, 'base64');

Unless the "'s around $.ContentID in the above "Parameters" that are the issue?

eikeon

comment created time in 23 days

Pull request review commentaws/aws-cdk

fix(stepfunctions): AttributeValue.withB

 export class DynamoAttributeValue {    * Sets an attribute of type Binary. For example:  "B": "dGhpcyB0ZXh0IGlzIGJhc2U2NC1lbmNvZGVk"    */   public withB(value: string) {-    this.attributeValue.B = value;+    this.attributeValue.B = Buffer.from(value, 'base64');

It's looking like it's probably on the receiving end arn:aws:states:::dynamodb:updateItem as CDK is generating "Parameters":{"Key":{"ContentID":{"B.$":"$.ContentID"}} as input to that.

eikeon

comment created time in 23 days

Pull request review commentaws/aws-cdk

fix(stepfunctions): AttributeValue.withB

 export class DynamoAttributeValue {    * Sets an attribute of type Binary. For example:  "B": "dGhpcyB0ZXh0IGlzIGJhc2U2NC1lbmNvZGVk"    */   public withB(value: string) {-    this.attributeValue.B = value;+    this.attributeValue.B = Buffer.from(value, 'base64');

This isn’t what we want. Looking at TaskInput and other bits now

eikeon

comment created time in 23 days

pull request commentaws/aws-cdk

fix(stepfunctions): AttributeValue.withB

And the "Full Docker Build" docker build -t aws-cdk . results in:

pkglint: success
pkglint: Done in 15.11s.
yarn-cling: yarn run v1.22.4
yarn-cling: $ npm run build && npm test
yarn-cling: > yarn-cling@0.0.0 build /app/tools/yarn-cling
yarn-cling: > tsc
yarn-cling: > yarn-cling@0.0.0 test /app/tools/yarn-cling
yarn-cling: > ln -sf ../../package2 test/test-fixture/package1/node_modules/ && jest
yarn-cling: ln: target ‘test/test-fixture/package1/node_modules/’ is not a directory: No such file or directory
yarn-cling: npm ERR! Test failed.  See above for more details.
yarn-cling: error Command failed with exit code 1.
yarn-cling: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
lerna ERR! yarn run build+test exited 1 in 'yarn-cling'

real	0m59.887s
user	1m21.266s
sys	0m6.455s
❌  Last command failed. Scroll up to see errors in log (search for '!!!!!!!!').
The command '/bin/sh -c ./build.sh ${BUILD_ARGS} && ./link-all.sh' returned a non-zero code: 1
eikeon

comment created time in 23 days

pull request commentaws/aws-cdk

fix(stepfunctions): AttributeValue.withB

I can not do a Full Clean Build It fails when running yarn build with

aws-cdk: Summary of all failing tests
aws-cdk: FAIL test/init.test.js
aws-cdk:   ● verify "future flags" are added to cdk.json
aws-cdk:     Could not add project AwsCdkTestRr27TV.csproj to solution AwsCdkTestRr27TV.sln. Error code: 127
aws-cdk:       at ChildProcess.<anonymous> (../../../../../private/var/folders/vy/d2wj3vw90fx06_p0s4tqndx40000gn/T/aws-cdk-testRR27tV/tmp/add-project.hook.ts:29:16)
aws-cdk: Test Suites: 1 failed, 28 passed, 29 total
aws-cdk: Tests:       1 failed, 140 passed, 141 total
aws-cdk: Snapshots:   0 total
aws-cdk: Time:        17.73s, estimated 43s
aws-cdk: Ran all test suites.
aws-cdk: Error: /Users/eikeon/aws-cdk/node_modules/jest/bin/jest.js --config jest.config.gen.json exited with error code 1
aws-cdk: Tests failed. Total time (18.7s) | /Users/eikeon/aws-cdk/node_modules/jest/bin/jest.js (18.7s)
aws-cdk: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
eikeon

comment created time in 23 days

pull request commentaws/aws-cdk

fix(stepfunctions): AttributeValue.withB

reading https://github.com/aws/aws-cdk/blob/master/CONTRIBUTING.md

eikeon

comment created time in 23 days

PR opened aws/aws-cdk

WIP: fix(stepfunctions): AttributeValue.withB

Commit Message

fix(stepfunctions): AttributeValue.withB

Fix type with conversion from base64

Close #7709

End Commit Message


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

+1 -1

0 comment

1 changed file

pr created time in 23 days

push eventeikeon/aws-cdk

Daniel Krech

commit sha d21382ed04967daca44b3a15e4207d8d7f546be2

WIP: Close #7724

view details

push time in 23 days

fork eikeon/aws-cdk

The AWS Cloud Development Kit is a framework for defining cloud infrastructure in code

https://aws.amazon.com/cdk

fork in 23 days

issue commentaws/aws-cdk

tasks.DynamoAttributeValue().withB double encoding value?

Adding the encoding='binary' positionally toString('binary') in the above workaround results in: AQERFArDrMOOacO0KWJPwpzDk8KEEAfCsFLCnMOQXsO3wp4 vs the expected AQERFArszmn0KWJPnNOEEAewUpzQXvee

closer... another couple characters. Perhaps the wrong flavor of base64 in this work around

eikeon

comment created time in 23 days

issue commentaws/aws-cdk

tasks.DynamoAttributeValue().withB double encoding value?

Will try to dive into CDK code to see how one might allow for passing in the already base64 encoded string to the withB

eikeon

comment created time in 23 days

issue commentaws/aws-cdk

tasks.DynamoAttributeValue().withB double encoding value?

I need to add encoding='binary' to the above toString for this to work. But then I get

{
  "error": "ReferenceError",
  "cause": {
    "errorType": "ReferenceError",
    "errorMessage": "encoding is not defined",
    "trace": [
      "ReferenceError: encoding is not defined",
      "    at eval (eval at handler (/var/task/index.js:8:12), <anonymous>:1:76)",
      "    at Runtime.handler (/var/task/index.js:8:12)",
      "    at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)"
    ]
  }
}

Which works here docker run --rm -it --entrypoint node -v "$PWD":/var/task:ro,delegated lambci/lambda:nodejs10.x -p "Buffer.from('AQERFArszmn0KWJPnNOEEAewUpzQXvee', 'base64').toString(encoding='binary')"

Guessing EvaluateExpression is trying to do something with the encoding= bit

eikeon

comment created time in 23 days

issue commentaws/aws-cdk

tasks.DynamoAttributeValue().withB double encoding value?

They are only starting with the right first few characters after that attempted work around.

For example (in one case) should be: AQERFArszmn0KWJPnNOEEAewUpzQXvee but is now: AQERFDDvv73vv70fNGXvv71V77+9ee+/vdeNBu+/ve+/vVLvv70Pbw==

eikeon

comment created time in 23 days

issue commentaws/aws-cdk

tasks.DynamoAttributeValue().withB double encoding value?

To work around this I'm taking the base64 encoded string (that is resulting from me passing in binary to the state machine without my base64 encoding it) and decoding it, and using the binary toString. For example, I run this extra task

        const contentIDB = new sfn.Task(this, "Decode ContentID", {
            task: new tasks.EvaluateExpression({
                expression:
                    "Buffer.from($.ContentID, 'base64').toString()",
            }),
            resultPath: "$.ContentIDB",
        });

and then use $.ContentIDB instead of $.ContentID

eikeon

comment created time in 23 days

issue commentaws/aws-cdk

RunLambdaTask with outputPath not working

Yep; the addition of a few .Payloads elsewhere in my step functions and no need for workarounds. So maybe just a note along side the deprecated notice of InvokeFunction of what options lie ahead. If I would have started with RunLambdaFunction I would not have been trying to preserve that exact behavior so hard

eikeon

comment created time in 23 days

issue commentaws/aws-cdk

RunLambdaTask with outputPath not working

Agreed re: a little clunky. Versus updating a few references elsewhere in the state machine to deal with the extra layer. I don't anticipate that's going to be a problem in my case.

eikeon

comment created time in 23 days

issue commentaws/aws-cdk

RunLambdaTask with outputPath not working

@shivlaks Thank you. And agreed re: very least since InvokeFunction is being deprecated and others will likely be going down this path with the rake on the ground.

From the CDK side, where higher level constructs are being created, would it be possible to add an option to RunLambdaTask that would create the extra Pass state?

Alternatively, document how one can expect an additional layer of output and update the rest of their state machine to expect it. Going to find out what that looks like now ;)

Thanks again for your help! Hope my feedback has been helpful as well. Let me know is there's anything I can do to help. Else I should be set now.

eikeon

comment created time in 24 days

issue commentaws/aws-cdk

RunLambdaTask with outputPath not working

I should have stared at https://docs.aws.amazon.com/step-functions/latest/dg/concepts-input-output-filtering.html longer. Had it in my head that the $outputPath happens before not after the $resultPath in that diagram.

But I'm still not able to replicate the TaskOutput I get with InvokeFunction when using RunLambdaTask. Attempt 1)

--- a/lib/getobject-statemachine.ts
+++ b/lib/getobject-statemachine.ts
@@ -48,6 +48,7 @@ export class GetObjectStateMachine extends cdk.Construct {

        const getObject = new sfn.Task(this, "Get Object", {
            task: new tasks.InvokeFunction(getObjectLambda, {}),
+            outputPath: "$.GetObjectResult.Payload",
            resultPath: "$.GetObjectResult",
        });

results in LambdaFunctionSucceeded with output:

{
  "output": {
    "Done": true,
    "Next": null,
    "BitsFetched": 0
  }
}

followed by ExecutionFailed with:

{
  "error": "States.Runtime",
  "cause": "An error occurred while executing the state 'Get Object' (entered at the event id #2). Invalid path '$.GetObjectResult.Payload' : No results for path: $['GetObjectResult']['Payload']"
}

Attempt 2)

--- a/lib/getobject-statemachine.ts
+++ b/lib/getobject-statemachine.ts
@@ -48,6 +48,7 @@ export class GetObjectStateMachine extends cdk.Construct {
 
         const getObject = new sfn.Task(this, "Get Object", {
             task: new tasks.InvokeFunction(getObjectLambda, {}),
+            outputPath: "$.GetObjectResult",
             resultPath: "$.GetObjectResult",
         });

followed by TaskStateExited with:

{
  "name": "Get Object",
  "output": {
    "Done": true,
    "Next": null,
    "BitsFetched": 0
  }
}

where as with the InvokeFunction the task state exited with:

{
  "name": "Get Object",
  "output": {
    "ParentContentID": "AQERFLelzPqbNGewZ3/CDNW/o/ZzuA4U",
    "Hash": "ERTg6jh13sV9G3ILH6qyChVfn4i2Bg==",
    "URL": "https://archive.org/download/snowyforluck00good/snowyforluck00good_metasource.xml",
    "Size": 0,
    "PartInfo": null,
    "GetObjectResult": {
      "Done": true,
      "Next": null,
      "BitsFetched": 0
    }
  }
}
eikeon

comment created time in 24 days

issue openedaws/aws-cdk

tasks.DynamoAttributeValue().withB double encoding value?

<!-- description of the bug: -->

I'm passing in a binary value to a step function which ends up getting base64 encoded somewhere along the way and it ends up getting base64 encoded a second time when I'm using withB as below. It's unclear from the current documentation if the string passed to withB should be base64 encoded already but it's type and the example seem to suggest it. If it's supposed to be a string of the raw binary is there a way decode an already base64 encoded string from the TaskInput.

The code I'm using to pass this value in as binary has been used in a couple other places using a couple different languages etc on the receiving end without issue. And no where am I knowingly encoding it a second time within the CDK state machine here.

withB's documentation shows

withB(value)🔹

public withB(value: string): DynamoAttributeValue
Parameters

value string
Returns

DynamoAttributeValue
Sets an attribute of type Binary.

For example: "B": "dGhpcyB0ZXh0IGlzIGJhc2U2NC1lbmNvZGVk"

Reproduction Steps

<!-- minimal amount of code that causes the bug (if possible) or a reference: -->

    new tasks.DynamoUpdateItem({
        tableName: props.content.tableName,
        partitionKey: {
            name: "ContentID",
            value: new tasks.DynamoAttributeValue().withB(
                sfn.TaskInput.fromDataAt("$.ContentID").value
            ),
        },
        ...

Error Log

<!-- what is the error message you are seeing? -->

Environment

  • CLI Version : 1.36.1 (build 4df7dac)
  • Framework Version: 1.36.1 (build 4df7dac)
  • OS : macOS Catalina 10.15.5 Beta (19F62f)
  • Language : Typescript

Other

<!-- e.g. detailed explanation, stacktraces, related issues, suggestions on how to fix, links for us to have context, eg. associated pull-request, stackoverflow, gitter, etc -->


This is :bug: Bug Report

created time in 24 days

issue commentaws/aws-cdk

Multiple ways to invoke Lambda function as a Step Functions task

The output of the Lambda is what's in the Payload I'm seeing in the Step Functions console under the TaskSucceeded event. Before the following event ExecutionFailed shows the above invalid path. I have both a outputPath: '$.Payload', and a resultPath: '$.GetObjectResult' if that matters? I can open a new issue in the morning if this is a bug, can't see what I'm doing wrong at this point. But it's the end of a long day.

{
  "resourceType": "lambda",
  "resource": "invoke",
  "output": {
    "ExecutedVersion": "$LATEST",
    "Payload": {
      "Done": true,
      "Next": null,
      "BitsFetched": 0
    },
    "SdkHttpMetadata": {
      "HttpHeaders": {
        "Connection": "keep-alive",
        "Content-Length": "41",
        "Content-Type": "application/json",
        "Date": "Thu, 30 Apr 2020 00:39:33 GMT",
        "X-Amz-Executed-Version": "$LATEST",
        "x-amzn-Remapped-Content-Length": "0",
        "x-amzn-RequestId": "dc8e5c92-28b4-41fd-a7bf-90970b93f51a",
        "X-Amzn-Trace-Id": "root=1-5eaa1e45-f6d040f0e5b2480a56f2dba4;sampled=0"
      },
      "HttpStatusCode": 200
    },
    "SdkResponseMetadata": {
      "RequestId": "dc8e5c92-28b4-41fd-a7bf-90970b93f51a"
    },
    "StatusCode": 200
  }
}
shivlaks

comment created time in 25 days

issue commentaws/aws-cdk

Multiple ways to invoke Lambda function as a Step Functions task

Upgraded to version 1.36.1 to switch to and test RunLambdaTask. I'm now able to get the right input to the lambda, but I'm not able to return the lambda's output as the task output via adding outputPath: "$.Payload", to the sfn.Task. Which the current overview has as an example. When doing so I get

{
  "error": "States.Runtime",
  "cause": "An error occurred while executing the state 'Get Object' (entered at the event id #2). Invalid path '$.Payload' : No results for path: $['Payload']"
}
shivlaks

comment created time in 25 days

issue commentaws/aws-cdk

Why are tasks.EvaluateExpression limited to expressions with containing state paths

for now I've worked around it with true ? Date.now().toString() : $.ContentID

eikeon

comment created time in a month

issue openedaws/aws-cdk

Why are tasks.EvaluateExpression limited to expressions with containing state paths

<!-- short description of the feature you are proposing: -->

Perhaps there's a good reason to limit EvaluateExpression to those containing state paths?

Use Case

<!-- why do you need this feature? -->

If not, I was just trying to use it in a state machine to create a timestamp via Date.now().toString()

Proposed Solution

<!-- Please include prototype/workaround/sketch/reference implementation: -->

Remove the constraint

Other

<!-- e.g. detailed explanation, stacktraces, related issues, suggestions on how to fix, links for us to have context, eg. associated pull-request, stackoverflow, gitter, etc -->

  • [x] :wave: I may be able to implement this feature request
  • [ ] :warning: This feature might incur a breaking change

This is a :rocket: Feature Request

created time in a month

issue commentaws/aws-cdk

Multiple ways to invoke Lambda function as a Step Functions task

@shivlaks Thank you. I'm all sorted with the possible exception of where to add documentation so other don't take the FIRE_AND_FORGET detour. Looks like the defaults of FIRE_AND_FORGET and RequestResponse etc leave RunLambdaTask acting like InvokeFunction did. Thanks again

shivlaks

comment created time in a month

issue commentaws/aws-cdk

Multiple ways to invoke Lambda function as a Step Functions task

When switching from InvokeFunction to RunLambdaTask w/ FIRE_AND_FORGET I see a TaskStarted with { "resourceType": "lambda", "resource": "invoke" } but it looks like the lambda is getting no input.

        const getObj = new sfn.Task(this, "Get Obj", {
            task: new tasks.InvokeFunction(getObjectLambda, {}),
        });

        const getObject = new sfn.Task(this, "Get Object", {
            task: new tasks.RunLambdaTask(getObjectLambda, {
                integrationPattern:
                    sfn.ServiceIntegrationPattern.FIRE_AND_FORGET,
            }),
            resultPath: '$.GetObjectResult',
        });

I also tried adding payload: sfn.TaskInput.fromDataAt('$') -- which looks like the default but still no input.

Did I mess up what the RunLambdaTask should look like to get it to ask like InvokeFunction (which is passed the tasks entire input)?

shivlaks

comment created time in a month

issue commentaws/aws-cdk

Multiple ways to invoke Lambda function as a Step Functions task

You're correct in your wondering, I didn't work out that FIRE_AND_FORGET would correspond to synchronous. I'll test that out now.

Also thrown off by the service integration not listing sync support for the lambda case here: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-service-integrations.html

shivlaks

comment created time in a month

issue commentaws/aws-cdk

Multiple ways to invoke Lambda function as a Step Functions task

@shivlaks, The RunLambdaTask does not appear to offer the sync running of a lambda that the InvokeFunction did. Has the initial offering by Step Function to (synchronously) invoke a Lambda function been deprecated? I've not seen any documentation suggesting that.

The use case I'm currently working on has a Lambda function that's meant to be run synchronously and modifying it to fit the service integration pattern (that doesn't support sync in the case of lambda) only complicates it. Is the intent to have RunLambdaTask also support sync?

shivlaks

comment created time in a month

issue commentaws/aws-cdk

Running "cdk diff" returns error code 1

Shouldn't the --fail option be removed if it doesn't actually work? Seems like it needs to be spelled cdk diff -c aws-cdk:enableDiffNoFail=true?

nicklaw5

comment created time in a month

more