profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/aidansteele/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Aidan Steele aidansteele Glass Echidna Melbourne, Australia http://www.glassechidna.com.au

issue commentturbot/steampipe-plugin-net

Add table net_certificate_logs

That's a great point regarding reliability and performance. You wouldn't want your rules to start failing because of third-party dependencies. Tricky 🤔

aidansteele

comment created time in 2 days

issue openedturbot/steampipe

Querying data-level services (e.g. Amazon Athena, S3 Select, etc)

(Sorry for not placing this in a specific plugin's issue board, I felt this was sufficiently different to warrant a top-level issue)

There are a handful of AWS services that can themselves support querying via APIs. A few examples:

  • AWS S3 Select can extract data from individual objects in S3 using SQL.

  • Amazon Athena allows you to write SQL queries that can query huge data sets across whole buckets, e.g. write SQL to query TBs of CloudTrail logs.

  • Both AWS RDS and Amazon Redshift have respective "Data APIs" where you can issue SQL queries through an AWS API without needing VPC connectivity.

  • Amazon Timestream can perform SQL queries on timeseries data.

It feels like it could be useful to be able to query these in Steampipe and join them with existing aws_* tables. But because they expose SQL interfaces, the current Steampipe table abstractions might not quite fit. Is this something you have already thought about and decided against? I suppose there is a chance that it doesn't align with your vision of the problems that Steampipe is intended to solve.

A concrete example of the value: right now I often use Amazon Athena to query CloudTrail logs to answer questions like "who created this bucket?". I might be asking that question because e.g. Steampipe told me that the bucket has public access. So it would be incredible to do something like (excuse my made-up SQL):

select
       s3.account_id,
       s3.name, 
       ct.userIdentity.principalArn 
from aws_s3_bucket s3, athena_cloudtrail ct 
where 
      bucket_policy_is_public = true 
  and ct.eventName = 'CreateBucket' 
  and ct.requestParameters.bucketName = s3.name

That would give me a list of people to chase up and find out if it was deliberately done that way, etc. Does that seem like it would make sense to add to Steampipe?

created time in 2 days

issue openedturbot/steampipe-plugin-net

Add table net_certificate_logs

General background / value proposition

Certificate transparency logs provide a publicly-accessible record of almost all SSL certificates issued on the Internet. It could be helpful to be able to query these tables from Steampipe to e.g. validate that no certificates have been issued for my domains that weren't in AWS ACM for one of my accounts.

Implementation questions

The question becomes: which data source should this query? A popular one is https://crt.sh. In fact, they even provide a publicly available Postgres instance (!!) for querying CT logs. You can get a sample of the SQL at this URL: https://crt.sh/?q=%25.steampipe.io&showSQL=y.

That page uses this SQL query:

WITH ci AS (
    SELECT min(sub.CERTIFICATE_ID) ID,
           min(sub.ISSUER_CA_ID) ISSUER_CA_ID,
           array_agg(DISTINCT sub.NAME_VALUE) NAME_VALUES,
           x509_commonName(sub.CERTIFICATE) COMMON_NAME,
           x509_notBefore(sub.CERTIFICATE) NOT_BEFORE,
           x509_notAfter(sub.CERTIFICATE) NOT_AFTER,
           encode(x509_serialNumber(sub.CERTIFICATE), 'hex') SERIAL_NUMBER
        FROM (SELECT *
                  FROM certificate_and_identities cai
                  WHERE plainto_tsquery('certwatch', 'steampipe.io') @@ identities(cai.CERTIFICATE)
                      AND cai.NAME_VALUE ILIKE ('%' || 'steampipe.io' || '%')
                  LIMIT 10000
             ) sub
        GROUP BY sub.CERTIFICATE
)
SELECT ci.ISSUER_CA_ID,
        ca.NAME ISSUER_NAME,
        ci.COMMON_NAME,
        array_to_string(ci.NAME_VALUES, chr(10)) NAME_VALUE,
        ci.ID ID,
        le.ENTRY_TIMESTAMP,
        ci.NOT_BEFORE,
        ci.NOT_AFTER,
        ci.SERIAL_NUMBER
    FROM ci
            LEFT JOIN LATERAL (
                SELECT min(ctle.ENTRY_TIMESTAMP) ENTRY_TIMESTAMP
                    FROM ct_log_entry ctle
                    WHERE ctle.CERTIFICATE_ID = ci.ID
            ) le ON TRUE,
         ca
    WHERE ci.ISSUER_CA_ID = ca.ID
    ORDER BY le.ENTRY_TIMESTAMP DESC NULLS LAST;

That yields (at the time of writing) these results: screenshot of psql

That presents us with a few questions:

  • Would you just use the Postgres->Postgres postgres_fdw with host crt.sh, user guest and DB certwatch?
  • Or should Steampipe attempt to simplify the (quite complex!) query above with some kind of view?
  • Maybe it's not worth using the postgres_fdw because it doesn't work with Steampipe's architecture, e.g. caching (I say this from a point of ignorance, I haven't looked into how that works)

Either way, I feel this could be useful. Thoughts?

created time in 2 days

issue openedturbot/steampipe

Interactive console becomes unresponsive after ";" query

Description: When I enter a line of only a semicolon into the interactive query console, it becomes unresponsive. Ctrl+C and Ctrl+D no longer work. The only option is to Ctrl+Z and then manually SIGKILL the steampipe and postgres processes.

Potential cause:

This code is executed when a query is only the semicolon character:

https://github.com/turbot/steampipe/blob/b7aa42ceb343b037a5cce2057bc048b3eb342f85/interactive/interactive_client.go#L411-L415

It appears to shortcircuit the typical query execution path, so the main REPL hangs here forever:

https://github.com/turbot/steampipe/blob/b7aa42ceb343b037a5cce2057bc048b3eb342f85/interactive/interactive_client.go#L121-L123

A workaround: Elsewhere in the file there is a workaround to return control back to the main REPL. Pasting this workaround (c.resultsStreamer.Done()) in the first code excerpt does appear to "fix" this bug:

https://github.com/turbot/steampipe/blob/b7aa42ceb343b037a5cce2057bc048b3eb342f85/interactive/interactive_client.go#L153-L155

Discussion: To be honest, I'm not 100% sure what the purpose of the "do nothing and restart interactive session" functionality is - so I can't submit a PR that I'm confident is the correct way to fix this. Hope this bug report helps nonetheless.

created time in 4 days

startedturbot/steampipe

started time in 4 days

push eventaidansteele/actions-stuff

Aidan Steele

commit sha d18247333521f8d3c270fd75b3e865b17a86b04a

Update bin.sh

view details

push time in 5 days

push eventaidansteele/actions-stuff

Aidan Steele

commit sha 1f09cc9dd82c972934cf36eb2e5304c2f7edf97e

Update blank.yml

view details

push time in 5 days

push eventaidansteele/actions-stuff

Aidan Steele

commit sha 99cbc41dcdfb7cb453b33750253186838841db2b

Update blank.yml

view details

push time in 5 days

push eventaidansteele/actions-stuff

Aidan Steele

commit sha 25e338a3f0206ddf8b917af9f98e4a11da0e7a48

Create bin.sh

view details

push time in 5 days

push eventaidansteele/actions-stuff

Aidan Steele

commit sha bf7207a3109162b938f380558817829932194618

Update blank.yml

view details

push time in 5 days

push eventaidansteele/actions-stuff

Aidan Steele

commit sha fb657094fb206be2edfb0c02ca91d4c36062f633

Update blank.yml

view details

push time in 5 days

push eventaidansteele/actions-stuff

Aidan Steele

commit sha 27025923cac9c00e3d301cc432d8e413cf34d432

Update blank.yml

view details

push time in 5 days

push eventaidansteele/actions-stuff

Aidan Steele

commit sha ad151e575e9cca74b42fa8c25fd91507d21105e1

Update blank.yml

view details

push time in 5 days

push eventaidansteele/actions-stuff

Aidan Steele

commit sha d32d6c43272ec1866422e86ce020c477e78fe8bf

Update blank.yml

view details

push time in 5 days

push eventaidansteele/actions-stuff

Aidan Steele

commit sha 79826ffe3446889bb4197f0ae06eb09868483517

Update blank.yml

view details

push time in 5 days

push eventglassechidna/dynamosize

Aidan Steele

commit sha f640f42aee4499013b511a9eabdcd0fd7195e6bb

hacks for life

view details

push time in 16 days

push eventglassechidna/dynamosize

Aidan Steele

commit sha 987da3243b106e97f45b726b21342e485c9a4c1a

no service-managed stacksets with transform :(

view details

push time in 16 days

push eventglassechidna/dynamosize

Aidan Steele

commit sha fa0706d9bc5b0aad99bcb4436f76afb7bb7c414d

fleshed out readme

view details

Aidan Steele

commit sha a8db1ad1abeba53e358819dd4bc348f183282ed4

readme note

view details

push time in 16 days

release glassechidna/dynamosize

0.1.0

released time in 16 days

delete tag glassechidna/dynamosize

delete tag : 0.1.0

delete time in 16 days

push eventglassechidna/dynamosize

Aidan Steele

commit sha b3c5f0c8bd3edd7027c0f830ab2cc69c696b6cf6

fix typo in ci script

view details

push time in 16 days

created tagglassechidna/dynamosize

tag0.1.0

AWS DynamoDB table sizes aren't reported to CloudWatch. This fixes that.

created time in 16 days

release glassechidna/dynamosize

0.1.0

released time in 16 days

push eventglassechidna/dynamosize

Aidan Steele

commit sha 1de890a185efd00455ec158a853b055161365b36

basic readme

view details

push time in 16 days

create barnchglassechidna/dynamosize

branch : main

created branch time in 16 days

created repositoryglassechidna/dynamosize

AWS DynamoDB table sizes aren't reported to CloudWatch. This fixes that.

created time in 16 days

push eventglassechidna/lambdaeip

Aidan Steele

commit sha a9f1b4cf58b83338c4e203e838e14554df8f3f34

update readme with SAR deployment

view details

push time in 17 days

created tagglassechidna/lambdaeip

tag0.1.0

Internet connectivity for your VPC-attached Lambda functions without a NAT Gateway

created time in 17 days

release glassechidna/lambdaeip

0.1.0

released time in 17 days

delete tag glassechidna/lambdaeip

delete tag : 0.1.0

delete time in 17 days