profile
viewpoint

mhart/alpine-node 2110

Minimal Node.js Docker Images built on Alpine Linux

mhart/dynalite 768

An implementation of Amazon's DynamoDB built on LevelDB

mhart/aws4 478

Signs and prepares Node.js requests using AWS Signature Version 4

mhart/aws4fetch 104

A compact AWS client for modern JS environments

mhart/awscred 33

Node.js module to resolve AWS credentials/region using env, ini files and IAM roles

jed/dynamo-client 26

A low-level client for accessing DynamoDB from node.js

mhart/dynamo-table 25

A lightweight module to map JS objects and queries to DynamoDB tables

mhart/awslogger 7

A CLI tool to send lines from stdin to AWS CloudWatch Logs

mhart/deep_learning_glossary 6

Simple, opinionated explanations of various things encountered in Deep Learning

jed/dynamo-sync 5

Differential data synchronization for Amazon's DynamoDB

issue commentmhart/aws4fetch

S3 console gives problematic URLs

Ah. Can you modify the hostname to the preferred format if you run into it? I kinda feel like it's better to not have an exception for this if AWS doesn't want ppl to use it anymore.

Liath

comment created time in 2 days

pull request commentmhart/aws4fetch

Improve TypeScript definition

I don't think I'll add dslint until they remove tslint and move to eslint

But happy to look at any jsdoc changes you'd like to make 👍

marcogrcr

comment created time in 2 days

push eventmhart/aws4

Michael Hart

commit sha 29b4fbf3e81c7ce4b053488c5f8cbf6bced39ad0

Document signQuery in README

view details

push time in 3 days

issue closedmhart/aws4

wish: Keep a changelog

For people considering whether or not to upgrade, a changelog is particularly helpful, and raw commit history is not a good substitute.

A number of project have taken to Github's "Release" feature as a convenient way to document changes. The downsidse to Github Releases is that it's not easily portable if the project ever leaves Github.

closed time in 3 days

markstos

issue commentmhart/aws4

wish: Keep a changelog

I've started using Github Releases and will continue to do so going forward – thanks for the nudge!

markstos

comment created time in 3 days

push eventmhart/aws4

Michael Hart

commit sha 5b354d635df53e0c5fc3b93d7a70dc3e542c9311

Remove package-lock

view details

push time in 3 days

push eventmhart/aws4

Michael Hart

commit sha 12a19a10fc87b9e93112ea6d948a4c1471770f99

Downgrade mocha/should again to support older node versions

view details

push time in 3 days

issue closedmhart/aws4

Old s3 path styles deprecated

s3 is not honoring the old path style for much longer (Deadline: Sept 2020)

it also seems that new buckets are not supported with the old style at all

Old

  • https://s3.amazonaws.com/<bucket>/<key>
  • https://s3-<region>.amazonaws.com/<bucket>/<key>

.

  • https://s3.amazonaws.com/jbarr-public/images/ritchie_and_thompson_pdp11.jpeg
  • https://s3-us-east-2.amazonaws.com/jbarr-public/images/ritchie_and_thompson_pdp11.jpeg

New

  • https://<bucket>.s3.amazonaws.com/<key>
  • https://jbarr-public.s3.amazonaws.com/images/ritchie_and_thompson_pdp11.jpeg

Reference

See https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/

closed time in 3 days

solderjs

issue commentmhart/aws4

Old s3 path styles deprecated

I've fixed the parsing aspect of this in v1.10.0

I think you're right that somehow constructing the host if the user has given the bucket in the path would be a bit problematic to stay backwards compatible, so I'm not going to support that.

I think from now on people will just need to specify the host when they want to query a particular bucket – aws4 should figure out the region and service correctly from there.

solderjs

comment created time in 3 days

release mhart/aws4

v1.10.0

released time in 3 days

push eventmhart/aws4

Michael Hart

commit sha ec6bb36564491e886082e1f5c0cca6dbd02a30a4

Add support for parsing newer style S3 hostnames

view details

Michael Hart

commit sha 90675bac4430088f171f2943e5a284876722699e

Cleanup README and examples

view details

Michael Hart

commit sha 09c55bf6d424a4b5547150f0809f854bd14f650a

1.10.0

view details

push time in 3 days

created tagmhart/aws4

tagv1.10.0

Signs and prepares node.js (and browser) requests using AWS Signature Version 4

created time in 3 days

push eventlambci/yumda

Erick Wendel

commit sha d9230fec223361b770744550e3da2f4027165966

chore(): add comment for packaging dependencies on nodejs example

view details

Michael Hart

commit sha d6023b54c12e6239222c2b75418d1e3fb0e7177d

Merge pull request #6 from ErickWendel/master feat(): add explicit zip folder on script nodejs after installing deps

view details

push time in 3 days

PR merged lambci/yumda

feat(): add explicit zip folder on script nodejs after installing deps

Signed-off-by: Erick Wendel erick.workspace@gmail.com

On Node.js apps was super hard to me figure out that to zip my dependencies I must use another flag. I put this on install.sh file to explicit it

+6 -0

3 comments

1 changed file

ErickWendel

pr closed time in 3 days

pull request commentlambci/yumda

feat(): add explicit zip folder on script nodejs after installing deps

Awesome, thanks for the contribution!

ErickWendel

comment created time in 3 days

push eventmhart/aws4fetch

Michael Hart

commit sha 11c00dbfa4c764207f7a4ade2219f5f525c75e66

Fix cloudflare secrets url in README

view details

push time in 3 days

push eventmhart/aws4fetch

Michael Hart

commit sha 9f20add7257f85cd5e661494ada0aeab36d9167c

Add resolve plugin to example

view details

push time in 3 days

pull request commentmhart/aws4fetch

Add a Typescript declaration file

Ugh, yeah the module name is a problem. Pity typescript didn't pick that up from any of the test within the repo itself.

Thanks for the PR @blakeembrey – this has been fixed in v1.0.9

Electroid

comment created time in 3 days

pull request commentmhart/aws4fetch

Generate declarations from `src/` directly

bike-shedding... sigh

Thanks for the quick fix – published as 1.0.9

blakeembrey

comment created time in 3 days

created tagmhart/aws4fetch

tagv1.0.9

A compact AWS client for modern JS environments

created time in 3 days

push eventmhart/aws4fetch

Michael Hart

commit sha 5db9030d9dc6818ce02f78067dd69cfeb16f17ac

1.0.9

view details

push time in 3 days

push eventmhart/aws4fetch

Blake Embrey

commit sha 84ba9b50e8f1d435a9edea2b6fcf94171d482ac7

Generate declarations from `src/` directly

view details

Michael Hart

commit sha 2d170d95703b831a3f85d7747b418f9563f23491

Merge pull request #17 from blakeembrey/be/fix-declaration-generation Generate declarations from `src/` directly

view details

push time in 3 days

PR merged mhart/aws4fetch

Generate declarations from `src/` directly

Generating with --outFile attempts to create a single bundle with declare module. However, you don't actually want declare module in TypeScript because two versions of the package will cause a global conflict.

+105 -107

1 comment

4 changed files

blakeembrey

pr closed time in 3 days

pull request commentmhart/aws4fetch

Generate declarations from `src/` directly

Gotcha. I changed the module name to aws4fetch and it worked – so I was gonna change the file name too. But I think this is a better solution 👍

blakeembrey

comment created time in 3 days

created tagmhart/aws4fetch

tagv1.0.8

A compact AWS client for modern JS environments

created time in 3 days

push eventmhart/aws4fetch

Michael Hart

commit sha b7f1ae16321845b45e247debdc36c36fc6e6fdb3

Fix module name - will see if I can get around renaming the file

view details

Michael Hart

commit sha c1816c216d296d24fe6eed991abcad17fd6f1abd

1.0.8

view details

push time in 3 days

push eventmhart/aws4fetch

Michael Hart

commit sha 37311c5a39f9bfb07d6223b3cd1605db00067af5

Fix example import

view details

push time in 3 days

PR closed mhart/aws4fetch

Improve TypeScript definition

WHAT?

  1. Fixed bugs in definitions:
    • Renamed AwsSigner to AwsV4Signer.
    • Removed retries and initRetryMs fields from: AwsSignerInit.
    • Removed body, headers, method, and url fields from: AwsRequestInit["aws"].
    • Made all fields optional in AwsRequestInit["aws"].
    • Removed invalid types (Blob, FormData, URLSearchParams, ReadableStream) from AwsSignerInit["body"].
    • Removed optional suffix (?) from required fields.
    • Removed redundant | null type in optional fields.
  2. Made all interface fields readonly.
  3. Added authHeader() and signature() to AwsV4Signer.
  4. Added jsdoc to the TypeScript definitions based on the README.md documentation.
  5. Added dtslint-based tests.

WHY?

  1. Allow the library to be used correctly with TypeScript.
  2. Make the interfaces immutable.
  3. Allow invocation of missing public methods.
  4. Make the library easier to use.
  5. Ensures types are correct and changes are backwards compatible.
+2131 -52

1 comment

6 changed files

marcogrcr

pr closed time in 3 days

pull request commentmhart/aws4fetch

Improve TypeScript definition

Hi @marcogrcr – thanks for the PR!

I went with the solution I said I was exploring here: https://github.com/mhart/aws4fetch/pull/7#issuecomment-631709172

I think this should cover all the issues you had. I'll close this, but happy to reopen if you think there's something missing

marcogrcr

comment created time in 3 days

pull request commentmhart/aws4fetch

Add a Typescript declaration file

I ended up implementing the JSDoc solution and have republished the types in 1.0.7

Let me know if there are any problems!

Electroid

comment created time in 3 days

created tagmhart/aws4fetch

tagv1.0.7

A compact AWS client for modern JS environments

created time in 3 days

push eventmhart/aws4fetch

Michael Hart

commit sha f1e0b7cedfd30a6c306f7c74032ad6e24338231a

Update notes about secrets

view details

Michael Hart

commit sha 325b5483da17f84b1f2a96fe311a12b2de1bb892

Run ESLint fixes

view details

Michael Hart

commit sha d78603b39babdf0e696a772c180a50623406438e

Add typing using JSDoc - update ESLint - add param tests - remove comments from output

view details

Michael Hart

commit sha 90b9f83e896babba589521d64de39b850385c9cd

1.0.7

view details

push time in 3 days

issue openedmicrosoft/TypeScript

Type 'ArrayBufferView' is not assignable to type 'DataView'

TypeScript Version: Nightly

<!-- Search terms you tried before logging this (so others can find this issue more easily) --> Search Terms:

ArrayBufferView

Expected behavior:

According to MDN, ArrayBufferView is one of these types:

  • Int8Array,
  • Uint8Array,
  • Uint8ClampedArray,
  • Int16Array,
  • Uint16Array,
  • Int32Array,
  • Uint32Array,
  • Float32Array,
  • Float64Array or
  • DataView.

However, in lib.es5.d.ts it's just an interface.

This would be fine if it were used consistently, but it appears that it's used in some places (eg BodyInit) but then in others the actual types from above are listed (eg crypto.subtle.digest). This means, for example, that going from BodyInit to crypto.subtle.digest (while filtering out incompatible types) is impossible.

According to these links:

  1. https://developer.mozilla.org/en-US/docs/Web/API/Request/Request#Syntax
  2. https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/digest#Syntax

This code should compile:

function bodyHash(body: BodyInit) {
    if (typeof body === 'string' ||
        body instanceof Blob ||
        body instanceof FormData ||
        body instanceof URLSearchParams ||
        body instanceof ReadableStream) {
            throw new Error('Only ArrayBuffer or ArrayBufferView allowed')
    }
    // error 2345: Type 'ArrayBufferView' is not assignable to type 'DataView'
    let result = crypto.subtle.digest('SHA-256', body)
}

Actual behavior:

Code fails to compile with type errors

<!-- Did you find other bugs that looked similar? --> Related Issues:

Possibly https://github.com/microsoft/TypeScript/issues/15402

Code

// https://developer.mozilla.org/en-US/docs/Web/API/ArrayBufferView
type ActualArrayBufferView = Int8Array | Uint8Array | Uint8ClampedArray | Int16Array | Uint16Array | Int32Array | Uint32Array | Float32Array | Float64Array | DataView

function actual(arr: ActualArrayBufferView) {
    // https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/digest#Syntax
    // works as it should with no error
    let result = crypto.subtle.digest('SHA-256', arr)
}

function builtin(arr: ArrayBufferView) {
    // error 2345: Type 'ArrayBufferView' is missing the following properties from type 'DataView': getFloat32, getFloat64, getInt8, getInt16, and 13 more
    actual(arr)

    // error 2345: Type 'ArrayBufferView' is not assignable to type 'DataView'
    let result = crypto.subtle.digest('SHA-256', arr)
}

// Real-world example
function bodyHash(body: BodyInit) {
    if (typeof body === 'string' ||
        body instanceof Blob ||
        body instanceof FormData ||
        body instanceof URLSearchParams ||
        body instanceof ReadableStream) {
            throw new Error('Only ArrayBuffer or ArrayBufferView allowed')
    }
    // error 2345: Type 'ArrayBufferView' is not assignable to type 'DataView'
    let result = crypto.subtle.digest('SHA-256', body)
}

<details><summary><b>Output</b></summary>

"use strict";
function actual(arr) {
    // https://developer.mozilla.org/en-US/docs/Web/API/SubtleCrypto/digest#Syntax
    // works as it should with no error
    let result = crypto.subtle.digest('SHA-256', arr);
}
function builtin(arr) {
    // error 2345: Type 'ArrayBufferView' is missing the following properties from type 'DataView': getFloat32, getFloat64, getInt8, getInt16, and 13 more
    actual(arr);
    // error 2345: Type 'ArrayBufferView' is not assignable to type 'DataView'
    let result = crypto.subtle.digest('SHA-256', arr);
}
// Real-world example
function bodyHash(body) {
    if (typeof body === 'string' ||
        body instanceof Blob ||
        body instanceof FormData ||
        body instanceof URLSearchParams ||
        body instanceof ReadableStream) {
        throw new Error('Only ArrayBuffer or ArrayBufferView allowed');
    }
    // error 2345: Type 'ArrayBufferView' is not assignable to type 'DataView'
    let result = crypto.subtle.digest('SHA-256', body);
}

</details>

<details><summary><b>Compiler Options</b></summary>

{
  "compilerOptions": {
    "noImplicitAny": true,
    "strictNullChecks": true,
    "strictFunctionTypes": true,
    "strictPropertyInitialization": true,
    "strictBindCallApply": true,
    "noImplicitThis": true,
    "noImplicitReturns": true,
    "useDefineForClassFields": false,
    "alwaysStrict": true,
    "allowUnreachableCode": false,
    "allowUnusedLabels": false,
    "downlevelIteration": false,
    "noEmitHelpers": false,
    "noLib": false,
    "noStrictGenericChecks": false,
    "noUnusedLocals": false,
    "noUnusedParameters": false,
    "esModuleInterop": true,
    "preserveConstEnums": false,
    "removeComments": false,
    "skipLibCheck": false,
    "checkJs": false,
    "allowJs": false,
    "declaration": true,
    "experimentalDecorators": false,
    "emitDecoratorMetadata": false,
    "target": "ES2017",
    "module": "ESNext"
  }
}

</details>

Playground Link: Provided

created time in 4 days

push eventlambci/docker-lambda

Michael Hart

commit sha c1487e9aa1f38ded7414f6a243fc898f598a9632

Update aws-sam-cli to 0.51.0

view details

push time in 4 days

issue commentmhart/aws4fetch

S3 console gives problematic URLs

More details here on this being a legacy format here: https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#s3-dash-region

Liath

comment created time in 4 days

issue commentmhart/aws4fetch

S3 console gives problematic URLs

Turns out it's deprecated – not sure why S3 Console still uses that for some regions?

https://twitter.com/Avarmax/status/1263281775331213314

Not sure whether I should add support for it or not, considering it's discouraged 😬

Liath

comment created time in 4 days

issue commentmhart/aws4fetch

S3 console gives problematic URLs

So weird, it doesn't do that for me.

I wonder if it's region specific?

Screen Shot 2020-05-20 at 7 05 34 PM

Liath

comment created time in 4 days

issue commentmhart/aws4fetch

S3 console gives problematic URLs

Can you show me where you encountered these URLs? I can't see them in the S3 Console anywhere

Liath

comment created time in 5 days

issue commentmhart/aws4fetch

Improvement request: Normalize certain HTTP methods to upper-case?

Sorry, forgot to comment here, but I fixed this back in 1.0.4 – thanks for reporting!

patrickjrm

comment created time in 5 days

pull request commentmhart/aws4fetch

Add a Typescript declaration file

Released as 1.0.6 – thanks again @Electroid for the contribution 🙌

Electroid

comment created time in 5 days

created tagmhart/aws4fetch

tagv1.0.6

A compact AWS client for modern JS environments

created time in 5 days

push eventmhart/aws4fetch

Michael Hart

commit sha 6c330faed8020e3db6415f5c5a9627d4705607df

1.0.6

view details

push time in 5 days

push eventmhart/aws4fetch

Ashcon Partovi

commit sha 3a436554a3180212b4e5b2667c3e099aec58bad8

Add a Typescript declaration file

view details

Michael Hart

commit sha 7d0abd4147de61d0d69a191225cc2a5d7807d867

Merge pull request #7 from Electroid/master Add a Typescript declaration file

view details

push time in 5 days

PR merged mhart/aws4fetch

Add a Typescript declaration file

Closes #6 by adding a Typescript declaration file.

+45 -0

7 comments

2 changed files

Electroid

pr closed time in 5 days

issue closedmhart/aws4fetch

TypeScript interfaces (definition files)

Could you please add interfaces for TypeScript? Currently everything is undefined/unknown.

closed time in 5 days

MickL

pull request commentmhart/aws4fetch

Add a Typescript declaration file

Hey all – sorry for the radio silence – tough times.

I actually wanted to get this happening with JSDoc comments, but I got sidetracked.

I'll get this in and then move to that – thanks!

Electroid

comment created time in 5 days

push eventlambci/docker-lambda

Michael Hart

commit sha 1fcfb22c3ef3705b5f40f993be2544813967df7f

Update .NET 3.1 SDK to 3.1.300 https://github.com/dotnet/core/blob/master/release-notes/3.1/3.1.4/3.1.300-sdk.md

view details

Michael Hart

commit sha 2fa7765f3f41bb4683eb1969a728914f3555d90f

Update Amazon.Lambda.Tools to 4.0.0 on dotnetcore3.1 runtime

view details

push time in 5 days

push eventlambci/docker-lambda

Michael Hart

commit sha 32315e5a08112da752af218655874a4ac4253cf0

Update aws-sam-cli to 0.50.0

view details

push time in 5 days

issue commentmhart/dynalite

UnexpectedParameter: Unexpected key 'TimeToLiveSpecification' found in params

It doesn't – but the error the poster is referring to is coming from validation in the aws-sdk – it's got nothing to do with this library

jseppi

comment created time in 5 days

push eventmhart/alpine-node

Michael Hart

commit sha 5bd241b24c986a2d863fafe5737cbb80a953283b

v14.3.0 full and slim builds

view details

push time in 6 days

created tagmhart/alpine-node

tagslim-14.3

Minimal Node.js Docker Images built on Alpine Linux

created time in 6 days

created tagmhart/alpine-node

tagslim-14.3.0

Minimal Node.js Docker Images built on Alpine Linux

created time in 6 days

created tagmhart/alpine-node

tag14.3

Minimal Node.js Docker Images built on Alpine Linux

created time in 6 days

created tagmhart/alpine-node

tag14.3.0

Minimal Node.js Docker Images built on Alpine Linux

created time in 6 days

issue commentmhart/dynalite

UnexpectedParameter: Unexpected key 'TimeToLiveSpecification' found in params

That error looks like it's coming from the aws-sdk?

jseppi

comment created time in 6 days

issue commentlambci/docker-lambda

It fails when python module OpenSSL is included.

Not sure I understand – can you explain the steps to reproduce?

rajeshpal007

comment created time in 6 days

push eventlambci/docker-lambda

Michael Hart

commit sha 33e48ce90b08a03fcfdd0b967dc2ba3e0f2d8549

Rearrange order of pip installs to try and reduce conflicts

view details

push time in 9 days

push eventlambci/docker-lambda

Michael Hart

commit sha 1016f89556130b3e095a10ed8300d2cd650daafd

Update aws-sam-cli to 0.49.0

view details

push time in 10 days

push eventlambci/yumda

Michael Hart

commit sha 531b3b18e35fdd382b0460603ce1b57aeb8df2dd

Update amzn2 packages

view details

Michael Hart

commit sha 4d501c9e74fe941c5195a920ba53d14b4cf41266

Update lambda2/kernel-headers to 4.14.177-139.254

view details

Michael Hart

commit sha aea2ff98ec3c7e56f43f5772bafb5eb0d7c459f8

Update lambda2/php to 7.3.17-2

view details

push time in 10 days

push eventlambci/yumda

Michael Hart

commit sha f8ebc6521453908649efa34e5d44ce788a3cdfcd

Update amzn1 packages - kernel - java-1.8.0-openjdk

view details

Michael Hart

commit sha 2268255c7f93cbf59cea694a3016554865e3d398

Update lambda1/kernel-headers to 4.14.177-107.254

view details

Michael Hart

commit sha 6b50572ff54e48feb79e3b515c3a3b468381758d

Update lambda1/java-1.8.0-openjdk-devel to 1:1.8.0.252.b09-2.51

view details

push time in 10 days

issue commentlambci/docker-lambda

Missing dependency chardet when using pipenv install on latest build-python3.7 image tag

I believe this should now be fixed with https://github.com/lambci/docker-lambda/commit/e69aa8b1f5ed85938e2be61db0ed9f574e17dd58 – I'm just unsetting PYTHONPATH. See https://github.com/lambci/docker-lambda/issues/272#issuecomment-628966591 if you want to revert to the older images that had this env var set.

samshelley

comment created time in 10 days

issue closedlambci/docker-lambda

Python 3.6 /var/runtime packages interfere with virtualenv packages

I'm getting python package dependency conflict when installing zappa into new virtual environment.

I use lambci/lambda-base:build-python3.6 image to build zappa project.

Steps to replicate:

docker run --rm -it lambci/lambda:build-python3.6 sh -c "virtualenv venv && source venv/bin/activate && pip install zappa && pip check" 

What's happening:

  • Zappa 0.48.2 requires python-dateutil<=2.7.0 dependency
  • virtualenv pip tries to install python-dateutil==2.6.1 to solve zappa dependency
  • virtualenv pip install fails to install python-dateutil==2.6.1 because
    Found existing installation: python-dateutil 2.7.3
    Not uninstalling python-dateutil at /var/runtime, outside environment /var/task/venv
    Can't uninstall 'python-dateutil'. No files were found to uninstall.
    
  • pip check fails with
    zappa 0.48.2 has requirement python-dateutil<2.7.0,>=2.6.1, but you have python-dateutil 2.7.3.
    
  • python-dateutil dependency isn't installed in venv and is injected into python runtime by PYTHONPATH from outside https://github.com/lambci/docker-lambda/blob/8f6145d649067704c8048578dae4b8e32063839f/python3.6/build/Dockerfile#L6

Question:

  • I would like to know why does lambci/lambda:build-python3.6 have extra python dependencies in image that interfere with every virtual environment?
  • Is it safe to ignore those extra global dependencies?

Some extra dependencies in /var/runtime installed here https://github.com/lambci/docker-lambda/blob/8f6145d649067704c8048578dae4b8e32063839f/python3.6/build/Dockerfile#L22

Workaround

I use workaround for this problem by resetting PYTHONPATH for now.

 docker run --rm -it -e PYTHONPATH="" lambci/lambda:build-python3.6 sh -c "virtualenv venv && source venv/bin/activate && pip install zappa && pip check"

closed time in 10 days

mojeto

issue commentlambci/docker-lambda

Python 3.6 /var/runtime packages interfere with virtualenv packages

I believe this should now be fixed with https://github.com/lambci/docker-lambda/commit/e69aa8b1f5ed85938e2be61db0ed9f574e17dd58 – I'm just unsetting PYTHONPATH. See https://github.com/lambci/docker-lambda/issues/272#issuecomment-628966591 if you want to revert to the older images that had this env var set.

mojeto

comment created time in 10 days

issue closedlambci/docker-lambda

New lambci:build-python3.7 image doesn't seem compatible with aws lambda us-west-2

Summary

When taking same code zip package created with latest lambci:build-python3.7 image. I am getting below exception inside us-west-2 deployments. This does not happen with us-east-1 deployments.

I believe this may be a bug with us-west-2 execution environment or us-east-1 is on some canary or something. I am raising an AWS ticket for this.

[ERROR] Runtime.ImportModuleError: Unable to import module 'redacted': No module named 'chardet'

Additional Notes

  1. When I downloaded the code zip from the lambda console us-east-1 and us-west-2 i see that there is a significant size difference in the code zips.
us-east-1 size: 9684109 bytes
us-west-2 size: 9333135 bytes
  1. If I take the downloaded west code zip and upload it to east; it will work fine.

  2. If i take the downloaded east code zip and upload it to west; it will also work fine.

  3. If i reupload the west code zip to west; it will resort back to the aformentioned error message.

Solution

For now I resorted back to an older copy I had of lambci:build-python3.7 I had pulled from 4 months ago and that is resolving my issue 100%.

I am wondering if this is related to some changes in https://github.com/lambci/docker-lambda/commit/4b7e20ab4d9bae5806428dda288bd6ad26db396d

closed time in 10 days

bclodius

issue commentlambci/docker-lambda

New lambci:build-python3.7 image doesn't seem compatible with aws lambda us-west-2

Ah, gotcha. Gonna close this as fixed by https://github.com/lambci/docker-lambda/commit/e69aa8b1f5ed85938e2be61db0ed9f574e17dd58

bclodius

comment created time in 10 days

issue commentlambci/docker-lambda

New lambci:build-python3.7 image doesn't seem compatible with aws lambda us-west-2

@bclodius you've pulled the latest build image? It should have this sha: https://hub.docker.com/layers/lambci/lambda/build-python3.7/images/sha256-e9046d2e685c9684cd5c1cc5eed078351042176c2a0fc7f73fe5bf487876ac3e

bclodius

comment created time in 10 days

push eventlambci/docker-lambda

Michael Hart

commit sha e69aa8b1f5ed85938e2be61db0ed9f574e17dd58

Remove PYTHONPATH from python build images This seems to interfere with virtualenv/pipenv and prevents it from installing packages if they already exist in /var/runtime * Fixes 271 * Fixes 272

view details

push time in 10 days

issue commentlambci/docker-lambda

New lambci:build-python3.7 image doesn't seem compatible with aws lambda us-west-2

Have pushed up new python build images that remove PYTHONPATH. The older images (with PYTHONPATH set) can be found at:

  • lambci/lambda:20200512-build-python2.7
  • lambci/lambda:20200512-build-python3.6
  • lambci/lambda:20200512-build-python3.7
  • lambci/lambda:20200512-build-python3.8

I'm hoping this won't break anyone's workflow (it should do the opposite) – although it may mean your bundles are slightly larger, but I think this should be a good thing™

bclodius

comment created time in 10 days

issue commentlambci/docker-lambda

New lambci:build-python3.7 image doesn't seem compatible with aws lambda us-west-2

Ah sorry, just read your msg again – if things were failing and then magically working, then I think you're right, they probably have been rolling back (or perhaps rolling out more widely beyond us-east-1)

bclodius

comment created time in 10 days

issue commentlambci/docker-lambda

New lambci:build-python3.7 image doesn't seem compatible with aws lambda us-west-2

I don't think it's a rollback per se – I think probably what's happening is that chardet has been added to /var/runtime in us-east-1, which the build image is based on, and therefore pipenv/virtualenv/whatever is deciding not to install it and so it doesn't get bundled up in your zip. This is fine in us-east-1, but not elsewhere where chardet doesn't exist.

Ideally there'd be a way to tell pipenv/virtualenv to always install packages even if they exist in PYTHONPATH, and that way it wouldn't matter if AWS adds/removes system libraries. I see it as a bug in pipenv/virtualenv to be honest, but can't see a good way around it.

So I think the answer has to be to not set PYTHONPATH in all of the build images, and ppl just end up installing (possibly) more packages than they may otherwise need to – but at least they'll be safe from changes on AWS' end

bclodius

comment created time in 10 days

issue commentlambci/docker-lambda

New lambci:build-python3.7 image doesn't seem compatible with aws lambda us-west-2

Basically, it seems that tools like pipenv, if they see that modules like chardet exist in /var/runtime (PYTHONPATH), then they don't install them – and I can't see an easy way to force them to do so?

bclodius

comment created time in 10 days

issue commentlambci/docker-lambda

New lambci:build-python3.7 image doesn't seem compatible with aws lambda us-west-2

Can you try lambci/lambda:build-python3.7-test and see if that fixes your issue?

I know it might be breaking change for some, but if that works I might just remove PYTHONPATH entirely from the build Dockerfile

bclodius

comment created time in 10 days

issue commentlambci/docker-lambda

New lambci:build-python3.7 image doesn't seem compatible with aws lambda us-west-2

I'm starting to think PYTHONPATH might be the issue here – it's set explicitly in the build images to match what happens in production – but it seems that certain tools like virtualenv and pipenv have issues with it when it's set.

bclodius

comment created time in 10 days

issue commentlambci/docker-lambda

New lambci:build-python3.7 image doesn't seem compatible with aws lambda us-west-2

Could it be a pipenv bug? https://github.com/pypa/pipenv/issues/3801

bclodius

comment created time in 11 days

issue commentlambci/docker-lambda

New lambci:build-python3.7 image doesn't seem compatible with aws lambda us-west-2

Perhaps related to https://github.com/lambci/docker-lambda/issues/271 ?

bclodius

comment created time in 11 days

push eventlambci/node-custom-lambda

Michael Hart

commit sha 294f12a468a9ae0c5490c67c6f47548ae2393fb6

Bump latest nodejs12 layer version

view details

push time in 11 days

push eventlambci/yumda

Michael Hart

commit sha 1c2e7f43042cdfc9a80af94b11c74af123037138

Add imagequant sources to vips spec

view details

push time in 11 days

issue commentmhart/aws4fetch

Not working when fetching object with DigitalOcean Spaces

No idea – there's no specific support for Digital Ocean here so you're on your own I'm afraid.

Are there no error messages to tell you what's wrong?

The only thing that looks weird to me is that you're manually messing with the URL, so maybe see if there's something around that? Perhaps try constructing the URL from scratch and see if that helps

jdelk

comment created time in 12 days

issue commentampproject/amphtml

Don't trigger amp-next-page for large viewports on initial load

Agreed, I think the only question is whether there are use cases that ppl want this behavior – eg, a related list of items that you're just paginating through. Because they're related, it might make sense to load as many as can fit in the viewport as once.

So basically, should there be an option for it

mhart

comment created time in 12 days

issue commentaws/aws-lambda-go

Does this project work with SNS destinations?

No probs, glad we got there in the end! And apologies again for my initial read of the issues you were seeing

DarthHater

comment created time in 13 days

issue openedampproject/amphtml

Don't trigger amp-next-page for large viewports on initial load

Describe the new feature or change to an existing feature you'd like to see

Hard to know whether this is a bug/feature-request that belongs in this project, or whether it's something that Googlebot should fix, but basically the way amp-next-page fetches multiple pages on initial load is apparently considered a no-no according to John Mueller:

https://twitter.com/JohnMu/status/1260120253469806593

I'd recommend not using infinite scrolling in this way when it comes to search. If these are separate, independent items, then expanding the viewport of a page shouldn't trigger them loading together. Mixing things on a single URL makes it possible that they're indexed together.

Describe alternatives you've considered

  • Only trigger page fetching if the user interacts with the page. Not sure if this is a hack, or the "correct" solution
  • Not using amp-next-page

created time in 13 days

issue commentaws/aws-lambda-go

Does this project work with SNS destinations?

Right, I believe that's the issue here. Kinesis, SQS and DynamoDB Streams all use synchronous invokes.

https://aws.amazon.com/blogs/architecture/understanding-the-different-ways-to-invoke-lambda-functions/

DarthHater

comment created time in 13 days

push eventlambci/docker-lambda

Michael Hart

commit sha 4b7e20ab4d9bae5806428dda288bd6ad26db396d

Update aws-lambda-builders to 0.9.0

view details

push time in 13 days

issue openedawslabs/serverless-application-model

Normalize casing of IAM actions

As I was debugging some permissions issues with one of my stacks, I noticed that the generated permissions sometimes had different casing/capitalization. It didn't take me too long to eliminate this as a cause of the issues I was seeing, but it still took me a bit of Googling to confirm.

Just for the sake of consistency, and so that no one else ends up debugging issues like this, it would be great to have the same casing over all permissions (and I think PascalCase seems to be the dominant standard here)

Here's an example of inconsistent casing (sns:publish here vs sns:Publish elsewhere), but there may be more:

https://github.com/awslabs/serverless-application-model/blob/37804045790dfa38a93e690e96e718b1d94e6dc9/samtranslator/model/iam.py#L71

created time in 13 days

issue commentaws/aws-lambda-go

Does this project work with SNS destinations?

FWIW, the following works for me.

SAM template.yaml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Resources:
  DestinationFunction:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: destination-test/
      Handler: destination-test
      Runtime: go1.x
      EventInvokeConfig:
        DestinationConfig:
          OnSuccess:
            Type: SNS
            Destination: !Ref DestinationTopic
          OnFailure:
            Type: SNS
            Destination: !Ref DestinationTopic

  DestinationTopic:
    Type: AWS::SNS::Topic

  DestinationSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      Endpoint: myemail@domain.com
      Protocol: email
      TopicArn: !Ref DestinationTopic

Outputs:
  DestinationFunction:
    Value: !GetAtt DestinationFunction.Arn

main.go:

package main

import (
	"github.com/aws/aws-lambda-go/lambda"
)

func handler(request interface{}) (string, error) {
	return "success", nil
}

func main() {
	lambda.Start(handler)
}

After sam build and sam deploy, I then do this:

aws lambda invoke --function-name <name> --invocation-type Event --payload '{}' /dev/stdout

And I receive an email along the lines of this:

{
  "version": "1.0",
  "timestamp": "2020-05-12T15:23:21.725Z",
  "requestContext": {
    "requestId": "d7a382da-dda9-46c7-85cc-9c1aa7b62f31",
    "functionArn": "arn:aws:lambda:us-east-1:<arn>:$LATEST",
    "condition": "Success",
    "approximateInvokeCount": 1
  },
  "requestPayload": {},
  "responseContext": {
    "statusCode": 200,
    "executedVersion": "$LATEST"
  },
  "responsePayload": "success"
}
DarthHater

comment created time in 13 days

issue commentaws/aws-lambda-go

Does this project work with SNS destinations?

Just confirming you're testing the Lambda with an asynchronous invocation (invocation type Event) and not a synchronous one (using the web console only trigger synchronous events)?

DarthHater

comment created time in 13 days

issue commentaws/aws-lambda-go

Does this project work with SNS destinations?

Ah, my bad – I couldn't see any SNS code in your sample

DarthHater

comment created time in 13 days

issue commentaws/aws-lambda-go

Does this project work with SNS destinations?

I'm guessing you want SNSEvent, not SQSEvent 😸

https://github.com/aws/aws-lambda-go/blob/master/events/README_SNS.md

DarthHater

comment created time in 13 days

issue commentlambci/docker-lambda

how to test the response of a node lambda

@dagda1 how'd you go with this?

dagda1

comment created time in 14 days

issue closedlambci/docker-lambda

Ruby Build Images Missing Rake

rake is not available in the build images, but it is available in the runtime images.

└─(16:31:49 on master)──> docker run -it --rm lambci/lambda:build-ruby2.5 sh                                    
sh-4.2# rake
Traceback (most recent call last):
	2: from /var/lang/bin/rake:23:in `<main>'
	1: from /var/lang/lib/ruby/site_ruby/2.5.0/rubygems.rb:294:in `activate_bin_path'
/var/lang/lib/ruby/site_ruby/2.5.0/rubygems.rb:275:in `find_spec_for_exe': can't find gem rake (>= 0.a) with executable rake (Gem::GemNotFoundException)

This is present in 2.5 and 2.7.

closed time in 14 days

dlahn

issue closedlambci/docker-lambda

npm install?

The readme suggests

# To compile native deps in node_modules
docker run --rm -v "$PWD":/var/task lambci/lambda:build-nodejs12.x npm rebuild

This is fine for compiling the lambda before pushing it.

But, how do I run a local version of my lambda with native dependencies? The function is going to be executed inside the container, so this

# Test a `handler` function from an `index.js` file in the current directory on Node.js v12.x
docker run --rm -v "$PWD":/var/task:ro,delegated lambci/lambda:nodejs12.x index.handler

is potentially going to fail if I've run the npm install from my host (thus defeating the purpose of using this system??). How are we supposed to get the native node modules inside the container during local development?

closed time in 14 days

sarink

issue commentlambci/docker-lambda

npm install?

Not really sure to be honest – the non-build images aren't intended for anything other than running Lambdas in them. Let us know if you find anything though

sarink

comment created time in 14 days

issue commentlambci/docker-lambda

Ruby Build Images Missing Rake

Fixed! Thanks for raising 👍

dlahn

comment created time in 14 days

pull request commentlambci/docker-lambda

Fix GEM_PATH to use correct Ruby installation

Got this in at https://github.com/lambci/docker-lambda/commit/efbb6a296f9cd7c5a84bf01b254d12b0f727d684 and have pushed up the latest ruby build images – thanks for the contribution!

I also added the /opt path back in so that ppl can still build with layers attached

dlahn

comment created time in 14 days

push eventlambci/docker-lambda

Dave Lahn

commit sha efbb6a296f9cd7c5a84bf01b254d12b0f727d684

Fix GEM_PATH to use correct Ruby installation

view details

Michael Hart

commit sha 84b8574e84b22fafb944b7328a1941320f7e2221

Add /opt gem path back in to ruby gem images

view details

push time in 14 days

issue closedlambci/docker-lambda

Missing dependency chardet when using pipenv install on latest build-python3.7 image tag

Hi @mhart, thanks so much for maintaining these dockerfiles!

We started running into an issue yesterday that corresponds roughly with when the latest tags for each build environment were posted. Basically, the chardet (https://pypi.org/project/chardet/) dependency which is relied on by many python libraries suddenly started vanishing from our build step. All of the other dependencies are present in site-packages, just not that one. Reverting back to an older sha (ie. FROM lambci/lambda@sha256:4d7db8d1724a0554574c506c8e5bcaa810718c91ccd9394d3e2456365108d56d) fixes the issue.

Here's a minimal repro: Dockerfile:

FROM lambci/lambda:build-python3.7

COPY Pipfile Pipfile.lock ./

RUN pipenv install --deploy

RUN cd $(pipenv --venv)/lib/python3.7/site-packages \
    && ls \
    && zip -qr9 /var/task/function.zip .

CMD echo done

Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
chardet = "*"

[requires]
python_version = "3.7"

Pipfile.lock

{
    "_meta": {
        "hash": {
            "sha256": "2a6cc4b02dd471cb1222b585553df91a14b96b0d50a569e011e1733cb1115fed"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "chardet": {
            "hashes": [
                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
            ],
            "index": "pypi",
            "version": "==3.0.4"
        }
    },
    "develop": {}
}

Then build it docker build . < Dockerfile and you will see the ls step does not print out the chardet dependency in the list of site packages.

If you swap out the FROM to be FROM lambci/lambda@sha256:4d7db8d1724a0554574c506c8e5bcaa810718c91ccd9394d3e2456365108d56d (a version one of us had from a few months ago) it fixes the problem.

We're still investigating to see if something else might be going on on our side, but any ideas on what might have changed recently would be immensely helpful.

closed time in 14 days

samshelley

issue commentlambci/docker-lambda

Missing dependency chardet when using pipenv install on latest build-python3.7 image tag

I don't have any digest history unfortunately – I only do explicit tags if there are (known) breaking changes. Will close this out, but let us know here if you solve your issues

samshelley

comment created time in 14 days

issue closedmhart/aws4

The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method

i'm getting AWS signature mismatch when i send request to API Gateway using axios

let data = JSON.stringify({
    "action": "message",
    "data": 'testing'
  })


let request = {
  host: '<API GATEWAY ID>.execute-api.<AWS REGION>.amazonaws.com',
  url: 'https://<API GATEWAY ID>.execute-api.<AWS REGION>.amazonaws.com/dev/@connections/<CONNECTION ID>',
  path: 'dev/@connections/<CONNECTION ID>',
  service: 'execute-api',
  region: '<AWS REGION>',
  data: data,
  body: data,
  headers: {
    'Content-Type': 'application/json'
  }
}

let signedRequest = aws4.sign(request,
                {
                  secretAccessKey: '<AWS SECRET KEY>',
                  accessKeyId: '<AWS ACCESS KEY>'
                })

let response = axios(signedRequest)

closed time in 14 days

bala-drg
more