Have you checked if there's any other process running on your machine that might be using the same port or blocking this? Sometimes cancelling on startup might avoid these services from being teared down. Additionally I would ask you to restart your machine to confirm.

Launching using docker-compose up localstack and DEBUG=1 in environment:

localstack_1             | Waiting for all LocalStack services to be ready
localstack_1             | 2021-01-18 20:51:01,864 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
localstack_1             | 2021-01-18 20:51:01,866 INFO supervisord started with pid 14
localstack_1             | 2021-01-18 20:51:02,870 INFO spawned: 'dashboard' with pid 20
localstack_1             | 2021-01-18 20:51:02,873 INFO spawned: 'infra' with pid 21
localstack_1             | 2021-01-18 20:51:02,876 INFO success: dashboard entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
localstack_1             | 2021-01-18 20:51:02,876 INFO exited: dashboard (exit status 0; expected)
localstack_1             | (. .venv/bin/activate; exec bin/localstack start --host)
localstack_1             | Starting local dev environment. CTRL-C to quit.
localstack_1             | LocalStack version: 0.12.4
localstack_1             | 2021-01-18 20:51:04,448 INFO success: infra entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
localstack_1             | Starting edge router (https port 4566)...
localstack_1             | Starting mock DynamoDB service on http port 4566 ...
localstack_1             | Starting mock DynamoDB Streams service on http port 4566 ...
localstack_1             | Starting mock Kinesis service on http port 4566 ...
localstack_1             | Starting mock S3 service on http port 4566 ...
localstack_1             | Listening at http://:::43967
localstack_1             | Initializing DynamoDB Local with the following configuration:
localstack_1             | Port:	38495
localstack_1             | InMemory:	false
localstack_1             | DbPath:	/mnt/data/dynamodb
localstack_1             | SharedDb:	true
localstack_1             | shouldDelayTransientStatuses:	false
localstack_1             | CorsParams:	*
localstack_1             | 
localstack_1             | 2021-01-18T20:51:05:DEBUG:bootstrap.py: Loading plugins - scope "services", module "localstack": <function register_localstack_plugins at 0x7f9c6f7dd550>
localstack_1             | [2021-01-18 20:51:06 +0000] [22] [INFO] Running on https://0.0.0.0:4566 (CTRL + C to quit)
localstack_1             | 2021-01-18T20:51:06:INFO:hypercorn.error: Running on https://0.0.0.0:4566 (CTRL + C to quit)
localstack_1             | 2021-01-18T20:51:06:INFO:localstack.multiserver: Starting multi API server process on port 57055
localstack_1             | [2021-01-18 20:51:06 +0000] [22] [INFO] Running on http://0.0.0.0:57055 (CTRL + C to quit)
localstack_1             | 2021-01-18T20:51:06:INFO:hypercorn.error: Running on http://0.0.0.0:57055 (CTRL + C to quit)
localstack_1             | Starting mock SQS service on http port 4566 ...
localstack_1             | Waiting for all LocalStack services to be ready
localstack_1             | Waiting for all LocalStack services to be ready
localstack_1             | Waiting for all LocalStack services to be ready
localstack_1             | Waiting for all LocalStack services to be ready
localstack_1             | Waiting for all LocalStack services to be ready
...

Hi @nhoughto, apologies for the delayed response. I was trying out on terraform 0.13 so unix_time was failing. I have fixed for terraform 0.14. https://github.com/localstack/localstack/pull/3482 should resolve the issue. Thanks.

• addresses https://github.com/localstack/localstack/issues/3475

Fixes https://github.com/localstack/localstack/issues/3009

Initializing the object SESBackend which is the default object for creating templates and listing them, transforming the input to using isoformat() for ISO 8601 (SDKv2 compatible) and collecting the output with the same ses_backend object.

Also some minor removals on what it seems to be legacy code.

Create:

$ aws --debug --endpoint-url=http://localhost:4566 --profile=test ses create-template --template 'TemplateName=hell-world,SubjectPart=subject,TextPart=hello\nworld,HtmlPart=hello<br/>world'
2021-01-12 03:31:59,466 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.18.211 Python/3.7.9 Linux/4.19.0-13-amd64 botocore/1.19.51
2021-01-12 03:31:59,466 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['--debug', '--endpoint-url=http://localhost:4566', '--profile=test', 'ses', 'create-template', '--template', 'TemplateName=hell-world,SubjectPart=subject,TextPart=hello\\nworld,HtmlPart=hello<br/>world']
2021-01-12 03:31:59,466 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x7fba58e79440>
2021-01-12 03:31:59,467 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7fba5937bef0>
2021-01-12 03:31:59,467 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fba59364950>
2021-01-12 03:31:59,467 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2021-01-12 03:31:59,470 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2021-01-12 03:31:59,470 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7fba59059b90>
2021-01-12 03:31:59,470 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/gattes/.miniconda3/lib/python3.7/site-packages/botocore/data/ses/2010-12-01/service-2.json
2021-01-12 03:31:59,478 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ses: calling handler functools.partial(<function _remove_commands at 0x7fba58f087a0>, commands_to_remove=['delete-verified-email-address', 'list-verified-email-addresses', 'verify-email-address'])
2021-01-12 03:31:59,478 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: delete-verified-email-address
2021-01-12 03:31:59,478 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: list-verified-email-addresses
2021-01-12 03:31:59,478 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: verify-email-address
2021-01-12 03:31:59,478 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ses: calling handler <function add_waiters at 0x7fba58e88560>
2021-01-12 03:31:59,484 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/gattes/.miniconda3/lib/python3.7/site-packages/botocore/data/ses/2010-12-01/waiters-2.json
2021-01-12 03:31:59,486 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('template', <awscli.arguments.CLIArgument object at 0x7fba58cf0c90>)])
2021-01-12 03:31:59,487 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ses.create-template: calling handler <function add_streaming_output_arg at 0x7fba58e79b90>
2021-01-12 03:31:59,487 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ses.create-template: calling handler <function add_cli_input_json at 0x7fba59369ef0>
2021-01-12 03:31:59,487 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ses.create-template: calling handler <function unify_paging_params at 0x7fba58efd5f0>
2021-01-12 03:31:59,496 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/gattes/.miniconda3/lib/python3.7/site-packages/botocore/data/ses/2010-12-01/paginators-1.json
2021-01-12 03:31:59,497 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ses.create-template: calling handler <function add_generate_skeleton at 0x7fba58f56a70>
2021-01-12 03:31:59,499 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ses.create-template: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7fba58cf0c50>>
2021-01-12 03:31:59,500 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ses.create-template: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fba58cf1490>>
2021-01-12 03:31:59,501 - MainThread - botocore.hooks - DEBUG - Changing event name from load-cli-arg.email.create-template.template to load-cli-arg.ses.create-template.template
2021-01-12 03:31:59,501 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ses.create-template.template: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fba593a4d10>
2021-01-12 03:31:59,502 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ses.create-template: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fba593a4f10>
2021-01-12 03:31:59,502 - MainThread - awscli.argprocess - DEBUG - Parsing param --template as shorthand
2021-01-12 03:31:59,502 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'TemplateName=hell-world,SubjectPart=subject,TextPart=hello\\nworld,HtmlPart=hello<br/>world' for parameter "template": {'TemplateName': 'hell-world', 'SubjectPart': 'subject', 'TextPart': 'hello\\nworld', 'HtmlPart': 'hello<br/>world'}
2021-01-12 03:31:59,502 - MainThread - botocore.hooks - DEBUG - Changing event name from load-cli-arg.email.create-template.cli-input-json to load-cli-arg.ses.create-template.cli-input-json
2021-01-12 03:31:59,502 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ses.create-template.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fba593a4d10>
2021-01-12 03:31:59,503 - MainThread - botocore.hooks - DEBUG - Changing event name from load-cli-arg.email.create-template.generate-cli-skeleton to load-cli-arg.ses.create-template.generate-cli-skeleton
2021-01-12 03:31:59,503 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ses.create-template.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fba593a4d10>
2021-01-12 03:31:59,503 - MainThread - botocore.hooks - DEBUG - Event calling-command.ses.create-template: calling handler <bound method CliInputJSONArgument.add_to_call_parameters of <awscli.customizations.cliinputjson.CliInputJSONArgument object at 0x7fba58cf0c50>>
2021-01-12 03:31:59,503 - MainThread - botocore.hooks - DEBUG - Event calling-command.ses.create-template: calling handler <bound method GenerateCliSkeletonArgument.generate_json_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fba58cf1490>>
2021-01-12 03:31:59,503 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2021-01-12 03:31:59,503 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2021-01-12 03:31:59,503 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2021-01-12 03:31:59,504 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2021-01-12 03:31:59,504 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2021-01-12 03:31:59,504 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/gattes/.miniconda3/lib/python3.7/site-packages/botocore/data/endpoints.json
2021-01-12 03:31:59,516 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fba599bad40>
2021-01-12 03:31:59,518 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.ses: calling handler <function add_generate_presigned_url at 0x7fba59a689e0>
2021-01-12 03:31:59,522 - MainThread - botocore.endpoint - DEBUG - Setting email timeout as (60, 60)
2021-01-12 03:31:59,523 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /home/gattes/.miniconda3/lib/python3.7/site-packages/botocore/data/_retry.json
2021-01-12 03:31:59,523 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: ses
2021-01-12 03:31:59,525 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.ses.CreateTemplate: calling handler <function generate_idempotent_uuid at 0x7fba599ebd40>
2021-01-12 03:31:59,527 - MainThread - botocore.hooks - DEBUG - Event before-call.ses.CreateTemplate: calling handler <function inject_api_version_header_if_needed at 0x7fba599755f0>
2021-01-12 03:31:59,527 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=CreateTemplate) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/1.18.211 Python/3.7.9 Linux/4.19.0-13-amd64 botocore/1.19.51'}, 'body': {'Action': 'CreateTemplate', 'Version': '2010-12-01', 'Template.TemplateName': 'hell-world', 'Template.SubjectPart': 'subject', 'Template.TextPart': 'hello\\nworld', 'Template.HtmlPart': 'hello<br/>world'}, 'url': 'http://localhost:4566/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7fba58a65110>, 'has_streaming_input': False, 'auth_type': None}}
2021-01-12 03:31:59,528 - MainThread - botocore.hooks - DEBUG - Event request-created.ses.CreateTemplate: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fba58a650d0>>
2021-01-12 03:31:59,528 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ses.CreateTemplate: calling handler <function set_operation_specific_signer at 0x7fba599eb290>
2021-01-12 03:31:59,529 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-01-12 03:31:59,529 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:localhost:4566
x-amz-date:20210112T023159Z

content-type;host;x-amz-date
b16c912c68ff17474ad84796b8cc27181b0b987089ffccf8f73fe3bab09f712b
2021-01-12 03:31:59,529 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20210112T023159Z
20210112/us-east-1/ses/aws4_request
7d00c8049b188304ece19bf3b071c788645e89247e374f9bc474edfd05d8b7d9
2021-01-12 03:31:59,529 - MainThread - botocore.auth - DEBUG - Signature:
7bb8c5e2c71971d81df114293680423750f76ce7717b7114480a86f80dcbdbed
2021-01-12 03:31:59,530 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=http://localhost:4566/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/1.18.211 Python/3.7.9 Linux/4.19.0-13-amd64 botocore/1.19.51', 'X-Amz-Date': b'20210112T023159Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=1234/20210112/us-east-1/ses/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=7bb8c5e2c71971d81df114293680423750f76ce7717b7114480a86f80dcbdbed', 'Content-Length': '175'}>
2021-01-12 03:31:59,531 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): localhost:4566
2021-01-12 03:31:59,557 - MainThread - urllib3.connectionpool - DEBUG - http://localhost:4566 "POST / HTTP/1.1" 200 217
2021-01-12 03:31:59,558 - MainThread - botocore.parsers - DEBUG - Response headers: {'content-type': 'text/html; charset=utf-8', 'content-length': '217', 'access-control-allow-origin': '*', 'access-control-allow-methods': 'HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH', 'access-control-allow-headers': 'authorization,content-type,content-md5,cache-control,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent,x-amz-target,x-amz-acl,x-amz-version-id,x-localstack-target,x-amz-tagging', 'access-control-expose-headers': 'x-amz-version-id', 'connection': 'close', 'date': 'Tue, 12 Jan 2021 02:31:59 GMT', 'server': 'hypercorn-h11'}
2021-01-12 03:31:59,558 - MainThread - botocore.parsers - DEBUG - Response body:
b'<CreateTemplateResponse xmlns="http://ses.amazonaws.com/doc/2010-12-01/"><CreateTemplateResult/><ResponseMetadata><RequestId>47e0ef1a-9bf2-11e1-9279-0100e8cf12ba</RequestId></ResponseMetadata></CreateTemplateResponse>'
2021-01-12 03:31:59,560 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ses.CreateTemplate: calling handler <botocore.retryhandler.RetryHandler object at 0x7fba58a65550>
2021-01-12 03:31:59,561 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2021-01-12 03:31:59,562 - MainThread - awscli.formatter - DEBUG - RequestId: 47e0ef1a-9bf2-11e1-9279-0100e8cf12ba

Output:

$ awslocal --profile=test ses list-templates
TEMPLATESMETADATA	2021-01-12T02:31:59.551596	hell-world

@whummer unfortunately the problem still persists. I could not apply the remote debugging in Eclipse:

Connection Type: Standard (Socket Attach) Host:127.0.0.1 Port:8000

Failed to connect to remote VM. Connection refused. Connection refused: connect

• Add support for EC2:Instance
• Fix issue with Fn:FindInMap
• Add integration test

Issues fixed: #2379 Can't update stack (aws cloudformation)

Coverage increased (+0.02%) to 69.891% when pulling c6f62417690dc874870d5c5b51181c714c52df32 on macnev2013:fix-cf-tags into 6926d47930a107de407534fb2321a3bb4c71075c on localstack:master.

Added

• Support for tags in cloudformation

Fixed

• 3476

i have done a bit of debug on the lambda_executor.py script and I see on line 55

whether to use our custom Java executor, or the default from lambci

TODO: deprecated, should be removed in the future

USE_CUSTOM_JAVA_EXECUTOR = False <<<< and then on line 301 we read java_opts if USE_CUSTOM_JAVA_EXECUTOR and is_java_lambda(runtime): # if running a Java Lambda with our custom executor, set up classpath arguments java_opts = Util.get_java_opts()

looks like java_opts never gets read in. if I remove the first condition and check for java_opts for any java lambda invoke, then I am able to bind to a given port.

Another issue in using debug_port once the above condition is fixed is, that the free tcp port it binds to is not output to user and user is unaware of which port to bind to for attaching the debugger.

<!-- Love localstack? Please consider supporting our collective: 👉 https://opencollective.com/localstack/donate -->

Type of request: This is a ...

[ x] bug report [ ] feature request

Detailed description

in docker compose under environment section adding LAMBDA_JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=:debug_port] or LAMBDA_JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=:5025

is not suspending the java execution of lambda to enable me to remote debug using IDE.

Expected behavior

should suspend execution, wait for remote debugger to attach to the process and then let developer step through the process.

Actual behavior

it ignores the option and just executes the lambda function.

Steps to reproduce

in docker compose under environment section adding LAMBDA_JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=:debug_port] or LAMBDA_JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=:5025

Command used to start LocalStack

docker_compose up which in turn starts localstack.

Client code (AWS SDK code snippet, or sequence of "awslocal" commands)

...

LocalStack version: 0.12.5

localstack_main | 2021-01-18T11:52:44:INFO:localstack.services.edge: Unable to find forwarding rule for host "localhost:4566", path "GET /favicon.ico", target header "", auth header "", data "b''"

In my case, I'm trying to simulate AWS S3 service

So gays, what will the resolution? if any at all?

Any updates on this? I haven't been able to use any version above 0.11.5 because of this. Any calls to SQS or DynamoDb from inside a lambda result in this error.

Here is my docker-compose file, with 0.11.5, if it helps:

services:
  localstack:
    image: localstack/localstack:0.11.5
    container_name: localstack
    restart: on-failure
    ports:
      - "4567-4584:4567-4584"
      - "4566:4566"
    environment:
      - SERVICES=sts,iam,lambda,sqs,apigateway,dynamodb,cloudwatch,ec2,ses,s3
      - LAMBDA_EXECUTOR=docker
      - LAMBDA_REMOTE_DOCKER=true
      - DOCKER_HOST=unix:///var/run/docker.sock
      - DEFAULT_REGION=eu-west-1
      - DEBUG=1
      - MAIN_CONTAINER_NAME=purchase_be_localstack
      - HOSTNAME=localstack
      - HOSTNAME_EXTERNAL=localstack
      - LAMBDA_DOCKER_NETWORK=purchase_be_api
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - api

  redis_opco:
    image: redis:6.0.5-alpine
    container_name: redis_opco
    restart: on-failure
    ports:
      - "6379:6379"
    networks:
      - api
  
  redis_auth:
    image: redis:6.0.5-alpine
    container_name: redis_auth
    restart: on-failure
    ports:
      - "6380:6379"
    networks:
      - api
  
networks:
  api:
    driver: bridge

@whummer Thank you. However, I have just tried with the latest: localstack/localstack@sha256:1890318c68b5f1325f2a329cb232c6dda34de63daf766ea081c9168773365171 and there is no change from the issue description above. The Expected behavior does not happen. Actual behavior is still what I'm experiencing.

a metadata key with a . is silently dropped:

Hi @shadowspawn, yes I missread that, my bad. If you have any findings whatsoever please post them here for further debugging. Cheers.

Thanks @mgagliardo. I'll try DEBUG=1 tomorrow.

The /mnt/data/ is the path inside the container I used for the docker managed volume, and does not exist with that path on the Mac host. (See the tail end of the example docker-compose.yml file for configuration.)

Some additional information from todays experimentation. On my Mac, localstack did eventually become ready after 6 min replaying 103 events. For my co-worker on a Windows PC, it took 91 minutes. (Not necessarily the same persistent data.)

Hey @shadowspawn , first of all, I was unable to reproduce this.

Can you please:

• Give a try to this using DEBUG=1 as an additional environment variable.
• On the other hand, /mnt/data/ does not exist on MacOS (I am using Catalina 10.15.7 as well) and can't be created as / is a RO filesystem, I am a bit worried it might be either that or permissions on your folder, can you please try something like creating a folder on i.e. /tmp/localstack, making sure your current user owns it and pointing DATA_DIR there?

Thanks!

Type of request: This is a ...

[x] bug report [ ] feature request

Detailed description

I am running localstack container using docker-compose with a volume mounted and DATA_DIR defined. I can get the multiple containers up and running from a clean configuration, but encounter problems after restarting. I encountered the problems when using services sqs+dynamodb.

Reproduced with localstack/localstack 0.12.5 and latest. macOS Catalina 10.15.7 Docker Engine v20.10.2

Please let me know if there is some additional logging I can supply, or if I should try for a minimal reproduction with a manual copy of a problematic recorded_api_calls.json file.

Expected behavior

Consistently able to launch services, multiple times.

Actual behavior

On the second launch, so the persistent data volume has contents, localstack does not come up and displays:

Waiting for all LocalStack services to be ready
Waiting for all LocalStack services to be ready
Waiting for all LocalStack services to be ready
...

Steps to reproduce

After the failure occurs (when I also had dynamodb enabled), the problem can be reproduced with just sqs and launching using a simple docker-compose.yml file:

version: '2.4'
services:
  localstack:
    image: localstack/localstack:latest
    environment:
      - SERVICES=sqs
      - DATA_DIR=/mnt/data
    ports:
      - '4566:4566'
    volumes:
      # Persistent data
      - type: volume
        source: localstack-data
        target: /mnt/data
volumes:
  localstack-data:

Launch with docker-compose up localstack loops indefinitely waiting for services to be ready.

Connecting to the container and deleting recorded_api_calls.json from DATA_DIR fixes the problem for the next launch.

Command used to start LocalStack

docker-compose up 

I am giving up on updating localstack for now, and will close this issue as I am going to pursue other approaches.

If anyone comes across this soon and has questions, I'm try and answer any questions while I still have the environment available.

I am not sure this was happening previously, but today I noticed the services do eventually start.

localstack_1             | Ready.
localstack_1             | 2021-01-18T01:55:44:INFO:localstack.utils.persistence: Restored 108 API calls from persistent file: /mnt/data/recorded_api_calls.json
localstack_1             | 2021-01-18T01:55:44:INFO:localstack.utils.analytics.profiler: Execution of "start_api_services" took 365855.30495643616ms

<!-- Love localstack? Please consider supporting our collective: 👉 https://opencollective.com/localstack/donate -->

Type of request: This is a ...

[x] bug report [ ] feature request

Detailed description

An S3 object metadata key with a . in it is lost silently.

Maybe the cause of this is similar to the cause of the underscore issue that was resolved with this workaround: https://github.com/localstack/localstack/pull/1965

Expected behavior

As on AWS, a metadata key with a . in it is written and read without issue:

aws s3api put-object --bucket my-bucket --key my-key --metadata key-with.dot=123143123
aws s3api head-object --bucket my-bucket --key my-key
{
    "AcceptRanges": "bytes",
    "LastModified": "2021-01-03T15:57:20+00:00",
    "ContentLength": 0,
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "ContentType": "binary/octet-stream",
    "Metadata": {
        "key-with.dot": "123143123"
    }
}

Actual behavior

Using localstack, a metadata key with a . is silently dropped:

aws --endpoint-url http://localhost:4566 s3api put-object --bucket my-bucket --key my-key --metadata key-with.dot=123143123
aws --endpoint-url http://localhost:4566 s3api head-object --bucket my-bucket --key my-key
{
    "LastModified": "2021-01-03T15:48:19+00:00",
    "ContentLength": 0,
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
    "ContentType": "binary/octet-stream",
    "Metadata": {}
}

Steps to reproduce

(included above)

Thanks for reporting @elrob . This should be fixed in the meantime - can you please pull the latest Docker image and give it another try? Please report here if the problem persists. Thanks

Type of request: This is a ...

[ X] bug report [ ] feature request

Detailed description

The current !Sub implementation only supports !Sub String. It doesn't support:

!Sub
  - String
  - Var1Name: Var1Value
    Var2Name: Var2Value

Details on how the sub intrinsic function behaves when given a variable map can be seen here: Intrinsic Function Reference Sub

Expected behaviour

The sub intrinsic function supports being provided a variable map.

Actual behaviour

The sub intrinsic function supports doesn't support being provided a variable map. The prevents resources which use this intrinsic function to not be created (e.g. in the below template, the SnsTopic resource is not created). The error outputted by localstack in debug mode is:

2020-12-09T13:44:16:ERROR:localstack.services.cloudformation.cloudformation_starter: Unable to parse and create resource "SnsTopic": replace() argument 2 must be str, not dict Traceback (most recent call last):
  File "/home/tobolov/linrepos/localstack/localstack/services/cloudformation/cloudformation_starter.py", line 365, in parse_and_create_resource
    result = _parse_and_create_resource(
  File "/home/tobolov/linrepos/localstack/localstack/services/cloudformation/cloudformation_starter.py", line 429, in _parse_and_create_resource
    resource_tuple = parsing.parse_resource_and_generate_name(logical_id, resource_json, resources_map)
  File "/home/tobolov/linrepos/localstack/.venv/lib/python3.8/site-packages/moto/cloudformation/parsing.py", line 282, in parse_resource_and_generate_name
    resource_tuple = parse_resource(resource_json, resources_map)
  File "/home/tobolov/linrepos/localstack/.venv/lib/python3.8/site-packages/moto/cloudformation/parsing.py", line 274, in parse_resource
    resource_json = clean_json(resource_json, resources_map)
  File "/home/tobolov/linrepos/localstack/localstack/services/cloudformation/cloudformation_starter.py", line 290, in clean_json
    rs = clean_json_orig(resource_json, resources_map)
  File "/home/tobolov/linrepos/localstack/.venv/lib/python3.8/site-packages/moto/cloudformation/parsing.py", line 202, in clean_json
    cleaned_val = clean_json(value, resources_map)
  File "/home/tobolov/linrepos/localstack/localstack/services/cloudformation/cloudformation_starter.py", line 290, in clean_json
    rs = clean_json_orig(resource_json, resources_map)
  File "/home/tobolov/linrepos/localstack/.venv/lib/python3.8/site-packages/moto/cloudformation/parsing.py", line 202, in clean_json
    cleaned_val = clean_json(value, resources_map)
  File "/home/tobolov/linrepos/localstack/localstack/services/cloudformation/cloudformation_starter.py", line 340, in clean_json
    resource_json['Fn::Sub'][0] = resource_json['Fn::Sub'][0].replace('${%s}' % k, v)
TypeError: replace() argument 2 must be str, not dict

Steps to reproduce

1. Deploy the following CloudFormation template:

---
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
  Environment:
    Type: String
    Default: 'companyname-ci'
Resources:
  SnsTopic:
    Type: AWS::SNS::Topic
    Properties: 
      TopicName: !Sub 
        - '${Env}-slack-topic'
        - { Env: !Select [1, !Split ["-" , !Ref Environment]] }

2. Run aws sns list-topics --endpoint-url http://localhost:4566 --region us-east-1
3. Observe that no topics exist, the topic listed in the CloudFormation template was not created.

Thanks for reporting @Tobolov . This should be fixed in #3470 - can you please pull the latest Docker image and give it another try? Thanks

Type of request: This is a ...

[ X] bug report [ ] feature request

Detailed description

The pseudo parameter AWS::Region doesn't work when nested inside the function Fn::GetAZs

Expected behavior

The pseudo parameter AWS::Region is resolved within the function Fn::GetAZs.

Actual behavior

2020-12-09T10:04:40:INFO:localstack.services.cloudformation.cloudformation_starter: Error on moto CF resource creation for PublicSubnetA. Ignoring, as should_be_created=False: 400 Bad Request:

<?xml version="1.0" encoding="UTF-8"?>
  <ErrorResponse>
    <Errors>
      <Error>
        <Code>InvalidParameterValue</Code>
        <Message>Value ({'Ref': 'AWS::Region'}a) for parameter availabilityZone is invalid. Subnets can currently only be created in the following availability zones: af-south-1a, af-south-1b, af-south-1c, ap-south-1a, ap-south-1b, eu-west-3a, eu-west-3b, eu-west-3c, eu-north-1a, eu-north-1b, eu-north-1c, eu-west-2a, eu-west-2b, eu-west-2c, eu-west-1a, eu-west-1b, eu-west-1c, ap-northeast-2a, ap-northeast-2a, ap-northeast-2c, ap-northeast-1a, ap-northeast-1c, ap-northeast-1d, ap-east-1a, ap-east-1b, ap-east-1c, sa-east-1a, sa-east-1c, ca-central-1a, ca-central-1b, ap-southeast-1a, ap-southeast-1b, ap-southeast-1c, ap-southeast-2a, ap-southeast-2b, ap-southeast-2c, eu-central-1a, eu-central-1b, eu-central-1c, eu-south-1a, eu-south-1b, eu-south-1c, us-east-1a, us-east-1b, us-east-1c, us-east-1d, us-east-1e, us-east-1f, us-east-2a, us-east-2b, us-east-2c, us-west-1a, us-west-1b, us-west-2a, us-west-2b, us-west-2c, me-south-1a, me-south-1b, me-south-1c, cn-north-1a, cn-north-1b, cn-northwest-1a, cn-northwest-1b, cn-northwest-1c, us-gov-west-1a, us-gov-west-1b, us-gov-west-1c, us-gov-east-1a, us-gov-east-1b, us-gov-east-1c.</Message>
        
      </Error>
    </Errors>
  <RequestID>7a62c49f-347e-4fc4-9331-6e8eEXAMPLE</RequestID>
</ErrorResponse>

Steps to reproduce

1. Deploy the following CloudFormation template to localstack:

---
AWSTemplateFormatVersion: '2010-09-09'

Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      EnableDnsSupport: true
      EnableDnsHostnames: true
      CidrBlock: "100.0.0.0/20"

  PublicNACL:
    Type: AWS::EC2::NetworkAcl
    Properties:
      VpcId:
        Ref: VPC
  PermitAnyFromPublic:
    Type: AWS::EC2::NetworkAclEntry
    Properties:
      CidrBlock: 0.0.0.0/0
      Egress: 'true'
      NetworkAclId:
        Ref: PublicNACL
      Protocol: '-1'
      RuleAction: allow
      RuleNumber: '100'
  PermitAnyToPublic:
    Type: AWS::EC2::NetworkAclEntry
    Properties:
      CidrBlock: 0.0.0.0/0
      NetworkAclId:
        Ref: PublicNACL
      Protocol: '-1'
      RuleAction: allow
      RuleNumber: '100'

  PublicSubnetA:
    Type: AWS::EC2::Subnet
    Properties:
      AvailabilityZone:
        Fn::Select:
        - 0
        - Fn::GetAZs:
            Ref: AWS::Region
      CidrBlock: "100.0.0.0/24"
      VpcId:
        Ref: VPC
  PublicSubnetANACLAssociation:
    Type: AWS::EC2::SubnetNetworkAclAssociation
    Properties:
      NetworkAclId:
        Ref: PublicNACL
      SubnetId:
        Ref: PublicSubnetA
  PublicSubnetARouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId:
        Ref: PublicRouteTableA
      SubnetId:
        Ref: PublicSubnetA
  PublicRouteTableA:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId:
        Ref: VPC

Outputs:
  PublicSubnetA:
    Value:
      Ref: PublicSubnetA
    Export:
      Name: 'public-sn-a'

2. Run aws cloudformation describe-stacks --endpoint-url http://localhost:4566 --region us-east-1.
3. Output for "ExportName": "public-sn-a" have value of None, should be id of created subnet.

Thanks for reporting @Tobolov . This should be fixed in #3470 - can you please pull the latest Docker image and give it another try? Thanks