That's a great point regarding reliability and performance. You wouldn't want your rules to start failing because of third-party dependencies. Tricky 🤔

aidansteele

comment created time in 2 days

issue openedturbot/steampipe

Querying data-level services (e.g. Amazon Athena, S3 Select, etc)

(Sorry for not placing this in a specific plugin's issue board, I felt this was sufficiently different to warrant a top-level issue)

There are a handful of AWS services that can themselves support querying via APIs. A few examples:

AWS S3 Select can extract data from individual objects in S3 using SQL.
Amazon Athena allows you to write SQL queries that can query huge data sets across whole buckets, e.g. write SQL to query TBs of CloudTrail logs.
Both AWS RDS and Amazon Redshift have respective "Data APIs" where you can issue SQL queries through an AWS API without needing VPC connectivity.
Amazon Timestream can perform SQL queries on timeseries data.

It feels like it could be useful to be able to query these in Steampipe and join them with existing aws_* tables. But because they expose SQL interfaces, the current Steampipe table abstractions might not quite fit. Is this something you have already thought about and decided against? I suppose there is a chance that it doesn't align with your vision of the problems that Steampipe is intended to solve.

A concrete example of the value: right now I often use Amazon Athena to query CloudTrail logs to answer questions like "who created this bucket?". I might be asking that question because e.g. Steampipe told me that the bucket has public access. So it would be incredible to do something like (excuse my made-up SQL):

select
       s3.account_id,
       s3.name, 
       ct.userIdentity.principalArn 
from aws_s3_bucket s3, athena_cloudtrail ct 
where 
      bucket_policy_is_public = true 
  and ct.eventName = 'CreateBucket' 
  and ct.requestParameters.bucketName = s3.name

That would give me a list of people to chase up and find out if it was deliberately done that way, etc. Does that seem like it would make sense to add to Steampipe?

created time in 2 days

issue openedturbot/steampipe-plugin-net

Add table net_certificate_logs

General background / value proposition

Certificate transparency logs provide a publicly-accessible record of almost all SSL certificates issued on the Internet. It could be helpful to be able to query these tables from Steampipe to e.g. validate that no certificates have been issued for my domains that weren't in AWS ACM for one of my accounts.

Implementation questions

The question becomes: which data source should this query? A popular one is https://crt.sh. In fact, they even provide a publicly available Postgres instance (!!) for querying CT logs. You can get a sample of the SQL at this URL: https://crt.sh/?q=%25.steampipe.io&showSQL=y.

That page uses this SQL query:

WITH ci AS (
    SELECT min(sub.CERTIFICATE_ID) ID,
           min(sub.ISSUER_CA_ID) ISSUER_CA_ID,
           array_agg(DISTINCT sub.NAME_VALUE) NAME_VALUES,
           x509_commonName(sub.CERTIFICATE) COMMON_NAME,
           x509_notBefore(sub.CERTIFICATE) NOT_BEFORE,
           x509_notAfter(sub.CERTIFICATE) NOT_AFTER,
           encode(x509_serialNumber(sub.CERTIFICATE), 'hex') SERIAL_NUMBER
        FROM (SELECT *
                  FROM certificate_and_identities cai
                  WHERE plainto_tsquery('certwatch', 'steampipe.io') @@ identities(cai.CERTIFICATE)
                      AND cai.NAME_VALUE ILIKE ('%' || 'steampipe.io' || '%')
                  LIMIT 10000
             ) sub
        GROUP BY sub.CERTIFICATE
)
SELECT ci.ISSUER_CA_ID,
        ca.NAME ISSUER_NAME,
        ci.COMMON_NAME,
        array_to_string(ci.NAME_VALUES, chr(10)) NAME_VALUE,
        ci.ID ID,
        le.ENTRY_TIMESTAMP,
        ci.NOT_BEFORE,
        ci.NOT_AFTER,
        ci.SERIAL_NUMBER
    FROM ci
            LEFT JOIN LATERAL (
                SELECT min(ctle.ENTRY_TIMESTAMP) ENTRY_TIMESTAMP
                    FROM ct_log_entry ctle
                    WHERE ctle.CERTIFICATE_ID = ci.ID
            ) le ON TRUE,
         ca
    WHERE ci.ISSUER_CA_ID = ca.ID
    ORDER BY le.ENTRY_TIMESTAMP DESC NULLS LAST;

That yields (at the time of writing) these results: screenshot of psql

That presents us with a few questions:

Would you just use the Postgres->Postgres postgres_fdw with host crt.sh, user guest and DB certwatch?
Or should Steampipe attempt to simplify the (quite complex!) query above with some kind of view?
Maybe it's not worth using the postgres_fdw because it doesn't work with Steampipe's architecture, e.g. caching (I say this from a point of ignorance, I haven't looked into how that works)

Either way, I feel this could be useful. Thoughts?

created time in 2 days

issue openedturbot/steampipe

Interactive console becomes unresponsive after ";" query

Description: When I enter a line of only a semicolon into the interactive query console, it becomes unresponsive. Ctrl+C and Ctrl+D no longer work. The only option is to Ctrl+Z and then manually SIGKILL the steampipe and postgres processes.

Potential cause:

This code is executed when a query is only the semicolon character:

https://github.com/turbot/steampipe/blob/b7aa42ceb343b037a5cce2057bc048b3eb342f85/interactive/interactive_client.go#L411-L415

It appears to shortcircuit the typical query execution path, so the main REPL hangs here forever:

https://github.com/turbot/steampipe/blob/b7aa42ceb343b037a5cce2057bc048b3eb342f85/interactive/interactive_client.go#L121-L123

A workaround: Elsewhere in the file there is a workaround to return control back to the main REPL. Pasting this workaround (c.resultsStreamer.Done()) in the first code excerpt does appear to "fix" this bug:

https://github.com/turbot/steampipe/blob/b7aa42ceb343b037a5cce2057bc048b3eb342f85/interactive/interactive_client.go#L153-L155

Discussion: To be honest, I'm not 100% sure what the purpose of the "do nothing and restart interactive session" functionality is - so I can't submit a PR that I'm confident is the correct way to fix this. Hope this bug report helps nonetheless.

created time in 4 days

startedturbot/steampipe

started time in 4 days