profile
viewpoint
James Kerr jameskerr Brim Security San Francisco https://looky.cloud Building UI at @brimsec

jameskerr/mini-db 2

Small database project for my data structures class at Chapman University

jameskerr/.dotfiles 0

Just the way I like it.

jameskerr/active_model-errors_details 0

Adds ActiveModel::Errors#details to return type of used validator - Backport from Rails 5.0

jameskerr/angular-file-upload 0

An AngularJS directive for file upload using HTML5 with FileAPI polyfill for unsupported browsers

jameskerr/angular-inarray-filter 0

Filter results that are contained inside an array

jameskerr/apns4ex 0

APNS for Elixir

jameskerr/bagpiper 0

My Bagpiping Website

jameskerr/beau 0

Beau's Website

PullRequestReviewEvent

issue commentbrimsec/brim

Offer "pcap info" summary within Brim

Great, I think that makes sense.

philrz

comment created time in 15 hours

Pull request review commentbrimsec/brim

refresh all connection spaces

 import {initSpace} from "../flows/initSpace" import Clusters from "../state/Clusters" import Current from "../state/Current" import getUrlSearchParams from "../lib/getUrlSearchParams"-import refreshSpaceNames from "../flows/refreshSpaceNames"+import refreshConnectionsSpaceNames from "../flows/refreshConnectionsSpaceNames"+import {globalDispatch} from "../state/GlobalContext"  export default async function(store: Store) {   const {space, host, port, id} = getUrlSearchParams()   global.windowId = id -  const clusterHost = host || "localhost"-  const clusterPort = port || "9867"-  const clusterId = `${clusterHost}:${clusterPort}`-  const cluster = {-    id: clusterId,-    host: clusterHost,-    port: clusterPort,-    username: "",-    password: ""+  const lastConn = Current.getConnection(store.getState())+  if ((port && host) || !lastConn) {+    const clusterHost = host || "localhost"+    const clusterPort = port || "9867"+    const clusterId = `${clusterHost}:${clusterPort}`+    const cluster = {+      id: clusterId,+      host: clusterHost,+      port: clusterPort,+      username: "",+      password: ""+    }++    store.dispatch(Clusters.add(cluster))

This conditional logic was getting hard to understand at first glance. What do you think about a version like this where we extract some of the connection setup into a new function?


const setupConnection = (host, port) => (dispatch, _, {globalDispatch}) => {
  const cluster = {
    host,
    port,
    id: [host, port].join(":"),
    username: "",
    password: ""
  }
  dispatch(Clusters.add(cluster))
  globalDispatch(Clusters.add(cluster))
  dispatch(Current.setConnectionId(cluster.id))
}

export default async function(store: Store) {
  const {space, host, port, id} = getUrlSearchParams()
  global.windowId = id

  const existingConnection = Current.getConnection(store.getState())

  if (host && port) {
    store.dispatch(setupConnection(host, port))
  } else if (!existingConnection) {
    store.dispatch(setupConnection("localhost", "9867"))
  }

  await store.dispatch(refreshConnectionsSpaceNames())

  const spaceId = space || Current.getSpaceId(store.getState())

  if (spaceId) store.dispatch(initSpace(spaceId))
}
mason-fish

comment created time in 15 hours

PullRequestReviewEvent

Pull request review commentbrimsec/brim

refresh all connection spaces

 export default (store: Store) => {     ipcRenderer.send(channel, getPersistable(store.getState()))   }) -  ipcRenderer.on("getState", (event, channel) => {-    ipcRenderer.send(channel, getPersistable(store.getState()))-  })-

Oh right 😅, thanks!

mason-fish

comment created time in 15 hours

PullRequestReviewEvent

issue openedbrimsec/brim

Update Log Post API to Multi-Part File Upload

As @mattnibs described in Slack:

The updated endpoint now expects a request in the following format. It will also be the desktop app's job to determine how to fill the progress bar, instead of relying on the response stream.

The zq branch to develop this with is in the PR: https://github.com/brimsec/zq/pull/1336

path: /space/:spaceid/logs
method: POST
query params: 
- stop_error <bool>: This instructs the server to abort the request if it receives a parsing error, or is unable to identify the content-type of a file. For brim I would recommend not using this param; the server will report these errors as warnings in the response body.
form data contents:
- name: "json_config"
  filename: <can be left blank>
  contents:  This should be a `json` payload of the typing configs, used to inform the zqd server how to richly encode data from an ndjson log file. This should be included as the first item in the FormData array.
- name: <can be left blank>
  filename: <name of the log file posted>
  contents: This should be the raw contents of a selected log file. You can post as many of these as you like.
response:
- content-type: application/json
- payload:
    - type <string>: Payload type. Will always be "LogPostResponse"
    - bytes_read <number>: The amount of bytes read from the form. The body of `json_type_config` will also be included in the count, so this might not match up the count of log data sent.
    - warnings <[]string>: A list of warnings encountered during log ingest.

created time in 2 days

PullRequestReviewEvent

Pull request review commentbrimsec/brim

refresh all connection spaces

 export default (store: Store) => {     ipcRenderer.send(channel, getPersistable(store.getState()))   }) -  ipcRenderer.on("getState", (event, channel) => {-    ipcRenderer.send(channel, getPersistable(store.getState()))-  })-

https://github.com/brimsec/brim/blob/master/src/js/electron/menu/appMenu.ts#L241

mason-fish

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentbrimsec/brim

refresh all connection spaces

 import {initSpace} from "../flows/initSpace" import Clusters from "../state/Clusters" import Current from "../state/Current" import getUrlSearchParams from "../lib/getUrlSearchParams"-import refreshSpaceNames from "../flows/refreshSpaceNames"+import refreshConnectionsSpaceNames from "../flows/refreshConnectionsSpaceNames"+import {globalDispatch} from "../state/GlobalContext"  export default async function(store: Store) {   const {space, host, port, id} = getUrlSearchParams()   global.windowId = id -  const clusterHost = host || "localhost"-  const clusterPort = port || "9867"-  const clusterId = `${clusterHost}:${clusterPort}`-  const cluster = {-    id: clusterId,-    host: clusterHost,-    port: clusterPort,-    username: "",-    password: ""+  const lastConn = Current.getConnection(store.getState())+  if ((port && host) || !lastConn) {+    const clusterHost = host || "localhost"+    const clusterPort = port || "9867"+    const clusterId = `${clusterHost}:${clusterPort}`+    const cluster = {+      id: clusterId,+      host: clusterHost,+      port: clusterPort,+      username: "",+      password: ""+    }++    store.dispatch(Clusters.add(cluster))

It looks like there is a case when we'll never get a cluster variable defined?

Is this the logic?

  1. If the host/port was provided in the url params, create a connection from them, add it, then set it to current.
  2. Else if there was a current connection already, do nothing more in this file
  3. Else create a default host and port, add it, then set it to current

Actually, do we always need to create the default one here?

mason-fish

comment created time in 2 days

Pull request review commentbrimsec/brim

refresh all connection spaces

 export default (store: Store) => {     ipcRenderer.send(channel, getPersistable(store.getState()))   }) -  ipcRenderer.on("getState", (event, channel) => {-    ipcRenderer.send(channel, getPersistable(store.getState()))-  })-

I am curious about the deletion here. It is used currently for gathering up all the window states when you click "Save Session for Testing Migrations".

mason-fish

comment created time in 2 days

Pull request review commentbrimsec/brim

refresh all connection spaces

+import {Thunk} from "../state/types"+import Clusters from "../state/Clusters"+import Spaces from "../state/Spaces"++export default function refreshConnectionsSpaceNames(): Thunk<Promise<void[]>> {

Since we are refreshing all the data about the space and not just the names anymore, a better name might be "refreshSpacesForAllConnections" ?

mason-fish

comment created time in 2 days

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

startedhovancik/stretchly

started time in 4 days

CommitCommentEvent
PullRequestReviewEvent
PullRequestReviewEvent

pull request commentbrimsec/brim

fix modal close

Here's a design I like for the loading state of a button: https://www.sketch.com/s/4b2957a2-910e-403c-9fda-617de055c061/a/OEY3dp#Inspector

image

mason-fish

comment created time in 8 days

Pull request review commentbrimsec/brim

fix modal close

 export default function ModalBox({name, children, ...props}: ModalBoxProps) {   }    return (-    <Animate show={active === name} enter={enter} exit="reverse">+    <Animate show={active === name} enter={enter}>

Yes, a faster "exiting" state is nice.

mason-fish

comment created time in 9 days

PullRequestReviewEvent

Pull request review commentbrimsec/brim

fix modal close

 export default function NewConnectionModal() {           return obj         }, {})         try {+          setIsFetching(true)           await dispatch(setConnection(toCluster({host})))         } catch (_) {           setErrors([{message: "Cannot connect to host"}])           return+        } finally {+          setIsFetching(false)         }         closeModal()       } else {         setErrors(form.getErrors())       }     },-    [f, config]+    [f, config, setIsFetching]

I don't think setIsFetching is necessary to include the deps here. I think the setters won't get stale like the values will.

mason-fish

comment created time in 9 days

Pull request review commentbrimsec/brim

fix modal close

 export default function NewConnectionModal() {           return obj         }, {})         try {+          setIsFetching(true)           await dispatch(setConnection(toCluster({host})))         } catch (_) {           setErrors([{message: "Cannot connect to host"}])           return+        } finally {+          setIsFetching(false)         }         closeModal()       } else {         setErrors(form.getErrors())       }     },-    [f, config]+    [f, config, setIsFetching]   )    const buttons = [     {label: "Cancel", click: (closeModal) => closeModal()},-    {label: "Connect", click: onSubmit}+    {label: "Connect", click: onSubmit, showSpinner: isFetching}

Instead of adding a showSpinner option, why not add a more generic "icon" option where you could inject the MacSpinner.

This does make me think about buttons with a loading state. I'd like to design a first-class "loading" state into the buttons. I always feel comforted as a user when I see those in other apps.

mason-fish

comment created time in 9 days

PullRequestReviewEvent
PullRequestReviewEvent

issue commentbrimsec/brim

Connection Detail view design

Here is a link to the connection detail designs: https://www.sketch.com/s/4b2957a2-910e-403c-9fda-617de055c061/p/connection-detail

mason-fish

comment created time in 9 days

Pull request review commentbrimsec/brim

1021 investigations space filter

+import {Thunk} from "../state/types"+import refreshSpaceNames from "./refreshSpaceNames"+import {getZealot} from "./getZealot"+import Current from "../state/Current"+import Investigation from "../state/Investigation"++const deleteSpaces = (ids: string[]): Thunk => (+  dispatch,+  getState,+  {globalDispatch}+) => {+  const zealot = dispatch(getZealot())+  const clusterId = Current.getConnectionId(getState())+  return Promise.all(

It'd be awesome if this could just be.

Promise.all(ids.map(id => dispatch(deleteSpace(id))))
  .catch(e => handle(e))
  .finally(() => dispatch(refreshSpaceNames()))
mason-fish

comment created time in 9 days

Pull request review commentbrimsec/brim

1021 investigations space filter

 import {State} from "../types"+import {Finding} from "./types" import {last} from "../../lib/Array" -export default {-  getInvestigation(state: State) {-    return state.investigation-  },+type Id = string | null -  getCurrentFinding(state: State) {-    return last(state.investigation)+export const getInvestigation = (connId: Id, spaceId: Id) => (

What do you think about a Current.getInvestigation() method as well, instead of having to get all these spaces and connection ids?

mason-fish

comment created time in 9 days

Pull request review commentbrimsec/brim

1021 investigations space filter

 import {Finding, InvestigationAction, InvestigationState} from "./types" import {SearchRecord} from "../../types" import {Ts} from "../../brim" import {last} from "../../lib/Array"+import produce from "immer" -const init: InvestigationState = []+const init = (): InvestigationState => ({}) -export default function reducer(-  state: InvestigationState = init,-  a: InvestigationAction-): InvestigationState {+export default produce((draft, a: InvestigationAction) => {

👍

mason-fish

comment created time in 9 days

Pull request review commentbrimsec/brim

1021 investigations space filter

 import {Thunk} from "../state/types" import refreshSpaceNames from "./refreshSpaceNames" import {getZealot} from "./getZealot"+import Current from "../state/Current"+import Investigation from "../state/Investigation" -const deleteSpace = (id: string): Thunk => (dispatch) => {+const deleteSpace = (id: string): Thunk => (+  dispatch,+  getState,+  {globalDispatch}+) => {   const zealot = dispatch(getZealot())+  const clusterId = Current.getConnectionId(getState())   return zealot.spaces.delete(id).then(() => {+    globalDispatch(Investigation.clearSpaceInvestigation(clusterId, id))     dispatch(refreshSpaceNames())

I think we ought to remove the refreshSpaces behavior from this thunk. Instead, we could add a Spaces.remove action that removes this individual one.

Then in your deleteSpaces function below, you could call this think in a loop.

mason-fish

comment created time in 9 days

Pull request review commentbrimsec/brim

1021 investigations space filter

 const StyledMagnifyingGlass = styled(MagnifyingGlass)`   } ` -type Props = {finding: Finding}+type Props = {finding: Finding; connId: string; spaceId: string} -export default React.memo<Props>(function FindingCard({finding}: Props) {+export default React.memo<Props>(function FindingCard({+  finding,+  connId,+  spaceId+}: Props) {   const dispatch = useDispatch()-  const clusterId = useSelector(Current.getConnectionId)-  const spaceIds = useSelector(Spaces.ids(clusterId))-  const findingSpaceName = get(finding, ["search", "spaceName"], "")    function onClick() {     dispatch(Search.restore(finding.search))     dispatch(submitSearch({history: true, investigation: false}))   } -  function renderWarning() {-    const findingSpaceId = get(finding, ["search", "spaceId"], "")-    const tip = "This space no longer exists"--    if (includes(spaceIds, findingSpaceId)) return null--    return (-      <div-        className="warning-body"-        data-tip={tip}-        data-effect="solid"-        data-place="right"-      >-        <Warning />-        <ReactTooltip />-      </div>-    )-  }-   const menu = usePopupMenu([     {       label: "Delete",-      click: () => globalDispatch(Investigation.deleteFindingByTs(finding.ts))+      click: () =>+        remote.dialog+          .showMessageBox({+            type: "warning",+            title: "Delete History Entry",+            message: `Are you sure you want to delete this history entry?`,+            buttons: ["OK", "Cancel"]+          })+          .then(({response}) => {+            if (response === 0)+              globalDispatch(+                Investigation.deleteFindingByTs(connId, spaceId, finding.ts)+              )+          })

Same comment as above. It would be awesome to consolidate the similar pieces of "FindingCard" and "FilterNode". This menu being an example similarity.

mason-fish

comment created time in 9 days

Pull request review commentbrimsec/brim

1021 investigations space filter

 function NodeRow({node, i}: Props) {     active: nodeIsActive(pinnedFilters, previous, node)   }) -  function renderWarning() {

Great! We don't need this any more.

mason-fish

comment created time in 9 days

Pull request review commentbrimsec/brim

1021 investigations space filter

 import FilterNode from "./FilterNode" import Investigation from "../state/Investigation" import Search from "../state/Search" import SearchBar from "../state/SearchBar"-import Spaces from "../state/Spaces/selectors"-import Warning from "./icons/warning-sm.svg" import usePopupMenu from "./hooks/usePopupMenu"+import {remote} from "electron" -type Props = {node: any; i: number}+type Props = {node: any; i: number; connId: string; spaceId: string} -function NodeRow({node, i}: Props) {+function NodeRow({node, i, connId, spaceId}: Props) {   const dispatch = useDispatch()   const pinnedFilters = useSelector(SearchBar.getSearchBarPins)   const previous = useSelector(SearchBar.getSearchBarPreviousInputValue)-  const clusterId = useSelector(Current.getConnectionId)-  const spaceIds = useSelector(Spaces.ids(clusterId))-  const findingSpaceName = get(-    node,-    ["data", "finding", "search", "spaceName"],-    ""-  )   const menu = usePopupMenu([     {       label: "Delete",       click: () => {-        const multiTs = node.mapChildren((node) => node.data.finding.ts)-        globalDispatch(Investigation.deleteFindingByTs(multiTs))+        remote.dialog

I think we decided at the last jam to only add a confirmation for "Delete All" history items. Confirming each one seems overly cautious.

mason-fish

comment created time in 9 days

PullRequestReviewEvent
PullRequestReviewEvent

issue commentbrimsec/brim

Query Library

I'll add that these queries should also be connection and time span independent. But what about pins? I'd think if a user saved a query with pins, we should retain the pins?

mason-fish

comment created time in 11 days

pull request commentbrimsec/brim

Move from Flow to Typescript

Fixes #1036

jameskerr

comment created time in 11 days

GollumEvent

push eventbrimsec/brim

James Kerr

commit sha 92bac6c79753e90a09237ea3a46b5eb89da3439a

Move from Flow to Typescript (#1075) * Move from Flow to Typescript * Remove last remnants of flow * Uncomment post install * bundle zealot with preset-env and fix quotes around extensions * Remove makefile from zealot

view details

push time in 11 days

delete branch brimsec/brim

delete branch : ts-rename

delete time in 11 days

PR merged brimsec/brim

Reviewers
Move from Flow to Typescript

Flow is gone and TypeScript is here!

Babel still transpiles the code, but instead of typechecking with npm run flow, we'll now use npm run tsc.

TypeScript caught more errors that Flow did, so I've fixed those it found.

All files must now end in .ts and all JSX files must end in .tsx

+37915 -67153

4 comments

1612 changed files

jameskerr

pr closed time in 11 days

push eventbrimsec/brim

James Kerr

commit sha a48511e594cd25f5dc65ca44aa7aaead89600245

Remove makefile from zealot

view details

push time in 11 days

push eventbrimsec/brim

Phil Rzewski

commit sha 20ddc6392176e001a7622c1a68aaddf9c5df9bd8

Point to Zeek v3.2.0-dev-brim9 (#1071)

view details

Brim Automation

commit sha 107f35cd6b79acff24db2e9151c0509b7dfd6de1

zq update through "Changelog for v0.21.0" by philrz This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1242, authored by @philrz, has been merged. Changelog for v0.21.0 https://github.com/brimsec/zq/compare/v0.20.0...e322b38

view details

Phil Rzewski

commit sha 9e1e32798645dc4e541910a66b949de432daa002

Update zq to v0.21.0 and prep v0.17.0 (#1073) https://github.com/brimsec/brim/compare/v0.16.0...107f35c

view details

Brim Automation

commit sha cb93b0ac007a6635f044ec0fc52321b7b7b89d93

zq update through "Simplify zng.fieldIter a little bit" by henridf This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1229, authored by @henridf, has been merged. Simplify zng.fieldIter a little bit

view details

Brim Automation

commit sha fc553554b60ba37366d52d12593118832045e34e

zq update through "change union implementation to conform with zng spec" by mccanne This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1245, authored by @mccanne, has been merged. change union implementation to conform with zng spec I noticed this problem when debugging the union column reader for zst. Union values were being incorrectly encoded by double encoding the length of the selector integer. This commit changes things to conform. We also noticed a bug in the ndjson reader where the union array values were being encoded as a sequence selector, value, selector, value instead of sequence of containers of selector/value pairs. This has been fixed.

view details

Brim Automation

commit sha 07cb7edf9cbc31d3771ceb6a36dd356f96e0f5d2

zq update through "simplify proc compiler" by mccanne This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1248, authored by @mccanne, has been merged. simplify proc compiler This commit simplifies the proc compiler so the nodes that can take multiple parents (sequential and parallel) are both factored out into separate methods. Fixes brimsec/zq#1169

view details

Brim Automation

commit sha b2ef0efb7acd1dae6e7a3480f0e835128de1a7b0

zq update through "Run Zeek system tests in Windows CI" by henridf This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1247, authored by @henridf, has been merged. Run Zeek system tests in Windows CI I found myself wanting this as I work on suricata integration and make changes to launchers/process management.

view details

Brim Automation

commit sha cb4ca37c9bf546f0371e74ed8f3133bb87df7f55

zq update through "cleanly separate flags and options in zio package" by mccanne This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1249, authored by @mccanne, has been merged. cleanly separate flags and options in zio package This commit eliminates significant technical debt for how we manage command flags and configuration options for the zio readers and writers. There were multiple patterns fighting agaist each other. This commit cleanly separates command flags from zio options and refactors the CLI tools to re-use many common flags. For example, you can now set the sortmem parameter and run profiling from most all of the tools. This also cleans up the way json type mappings are read and configured into the reader options. In the new approach, each zio reader/writer is responsible for creating an options struct and package zio/options glues them together. The only package under zio that knows about command flags is zio/flags, and its job is to relate flags to configuration options. We also deleted the "types" output format rather than make it work here since it was hokey (my bad) and I think a better approach is to make proc that helps collect up all the types. Whether and when we switch over to cobra (or something else), these changes are orthogonal and still very helpful. Fixes brimsec/zq#1250

view details

Brim Automation

commit sha 2373fe3e10cb9303cf244502c08e134f76a511d4

zq update through "move zng flattener from zeekio to zng/flattener" by mccanne This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1252, authored by @mccanne, has been merged. move zng flattener from zeekio to zng/flattener This commit moves the flattening logic out of the zeekio package and into its own package under zng since this operation is a generic zng transformation and is not specific to zio. This will be helpful also to the csv writer when we add it.

view details

Brim Automation

commit sha 78d10e90df590ba6d58dddf7cef15f1ce0a2988e

zq update through "choose better names for zio options and flags" by mccanne This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1254, authored by @mccanne, has been merged. choose better names for zio options and flags @nwt was right... these are better names. Fixes brimsec/zq#1253

view details

Brim Automation

commit sha 4ecceaba9ea2ea8b72a15e56d44440d3fae448c9

zq update through "simplify zng visitor" by mccanne This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1258, authored by @mccanne, has been merged. simplify zng visitor I was working on updating zng sets and didn't understand why I had to change so many places in zng/walk.go so I simplified it. This commit simplifies the code in zng/walk.go to use a common function for descending into container types. The error messages are a little different and a union record value gets visited now instead of omitted, but I think that was a bug/oversight. We also changed zng.RecordVisitor to zng.Visitor since you can now called zng.Walk on any type.

view details

Brim Automation

commit sha d936bad8d2cdd06d137b0b118a5514227ff9e375

zq update through "fix zqd version breakage" by mccanne This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1256, authored by @mccanne, has been merged. fix zqd version breakage This commit fixes a bug introduced by the recent refactoring of cli flags and zio options. While we're at it, we added a zapi version command and added a test to make sure it's working. We also simplified the format of the version response message. The brim app doesn't seem to use the /version endpoint so the change shouldn't break anything. Fixes brimsec/zq#1255

view details

Brim Automation

commit sha 9e5830325027dee4004aa2c898606073ae8f2ad9

zq update through "add csv writer" by mccanne This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1267, authored by @mccanne, has been merged. add csv writer Fixes brimsec/zq#1237

view details

James Kerr

commit sha 1a7dccdc64ffd0ac4ee0750b62a8384dfe90c982

Move from Flow to Typescript

view details

James Kerr

commit sha aa477a70e276ff8e1d246f0c7cabf8e5fe56fb10

Remove last remnants of flow

view details

James Kerr

commit sha e4d28a1b21b66d0c3610a9ea8d800dac48a5a172

Uncomment post install

view details

James Kerr

commit sha 64952587a196bbeda62a2817f5ed900ea4e038d8

bundle zealot with preset-env and fix quotes around extensions

view details

push time in 12 days

push eventbrimsec/brim

James Kerr

commit sha 8c292962f96aaf655f30966aa8ed64a5ffb1bfb1

bundle zealot with preset-env and fix quotes around extensions

view details

push time in 12 days

issue commentbrimsec/brim

Support import of nanosecond pcap files

Thanks @philrz !

philrz

comment created time in 12 days

issue commentbrimsec/zq

tidy up zjson

@mccanne sorry for the radio silence. Thanks for revisiting the zjson format again. I think your proposals all make sense. I forgot about the array/set/union encoding issues, so I'm glad those were discovered again and issues made.

I think the current zjson model is great for all the reasons you've mentioned above. These new ergonomic changes will make the data even more portable.

As a user, if I want rich types, streaming responses, and no-bloat payloads, I download a library (zealot, python, zapi) and I'm on my way. If I want a quick and dirty fetch to do an ad hoc chart on https://observablehq.com/, codepen, or curl | jq, then I am able to do that too. I dig it.

mccanne

comment created time in 12 days

pull request commentbrimsec/brim

Move from Flow to Typescript

@alfred-landrum thanks for doing the windows debugging. I'll see if I can get babel to make those files.

jameskerr

comment created time in 12 days

issue commentbrimsec/brim

remove usage of make in package.json

Yup, this makes sense.

alfred-landrum

comment created time in 12 days

delete branch brimsec/brim

delete branch : ts

delete time in 14 days

PR closed brimsec/brim

TypeScript
+39210 -65837

0 comment

1590 changed files

jameskerr

pr closed time in 14 days

push eventbrimsec/brim

James Kerr

commit sha d8fa979320039c64fd19074af9fc7dab536fa06b

Uncomment post install

view details

push time in 14 days

push eventbrimsec/brim

James Kerr

commit sha 300dc4ee666324a11629cca60cd9c0304cfbf946

Remove last remnants of flow

view details

push time in 14 days

PR opened brimsec/brim

Move from Flow to Typescript
+37580 -66015

0 comment

1588 changed files

pr created time in 14 days

create barnchbrimsec/brim

branch : ts-rename

created branch time in 14 days

create barnchbrimsec/brim

branch : ts-rebase

created branch time in 14 days

PR opened brimsec/brim

TypeScript
+39210 -65837

0 comment

1590 changed files

pr created time in 14 days

create barnchbrimsec/brim

branch : ts

created branch time in 14 days

PullRequestReviewEvent

Pull request review commentbrimsec/brim

refactor to usePopupMenu

 export default function SavedSpacesList({spaces, spaceContextMenu}: Props) {     <menu className="saved-spaces-list">       {spaces         .sort((a, b) => (a.name > b.name ? 1 : -1))-        .map(brim.space)-        .map((s) => {-          const progress = s.ingesting() && (-            <div className="small-progress-bar">-              <ProgressIndicator percent={s.ingestProgress()} />-            </div>-          )-          return (-            <li key={s.id}>-              <a-                href="#"-                onClick={onClick(s.id)}-                onContextMenu={() => {-                  !s.ingesting() &&-                    showContextMenu(spaceContextMenu(s.id, s.name))-                }}-                className={classNames("space-link", {-                  "current-space-link": s.id === currentSpaceId-                })}-              >-                <NameWrap>-                  <SpaceIcon type={s.getType()} className="space-icon" />-                  <span className="name">{s.name}</span>-                </NameWrap>-                {progress}-              </a>-            </li>-          )+        .map((space) => {+          return <SpaceListItem key={space.id} space={space} />         })}     </menu>   ) }++const SpaceListItem = ({space}: {space: Space}) => {+  const dispatch = useDispatch()+  const clusterId = useSelector(Current.getConnectionId)+  const currentSpaceId = useSelector(Current.getSpaceId)+  const s = brim.space(space)++  const onClick = (e) => {+    e.preventDefault()+    dispatch(initSpace(s.id))+  }+  const contextMenu = usePopupMenu([+    {+      label: "Rename",+      click: () => {+        dispatch(Modal.show("space", {clusterId, spaceId: s.id}))+      }+    },+    {+      label: "Delete",+      click: () => {+        remote.dialog+          .showMessageBox({+            type: "warning",+            title: "Delete Space",+            message: `Are you sure you want to delete ${s.name}?`,+            buttons: ["OK", "Cancel"]+          })+          .then(({response}) => {+            if (response === 0) dispatch(deleteSpace(s.id))+          })+      }+    }+  ])+  const progress = s.ingesting() && (+    <div className="small-progress-bar">+      <ProgressIndicator percent={s.ingestProgress()} />+    </div>+  )+  return (+    <li>+      <a+        href="#"+        onClick={onClick}+        onContextMenu={() => {+          !s.ingesting() && contextMenu.onClick()

I think you could call contextMenu.open() here instead of modifying the onClick handler below.

mason-fish

comment created time in 17 days

Pull request review commentbrimsec/brim

refactor to usePopupMenu

 import {getZealot} from "./getZealot"  export default (clusterId: string, spaceId: string, name: string): Thunk => (   dispatch,-  getState+  getState,+  {globalDispatch} ) => {   const state = getState()   const zealot = dispatch(getZealot())   const tabs = Tabs.getData(state)    return zealot.spaces.update(spaceId, {name}).then(() => {-    dispatch(Spaces.rename(clusterId, spaceId, name))+    globalDispatch(Spaces.rename(clusterId, spaceId, name))

Good catch

mason-fish

comment created time in 17 days

PullRequestReviewEvent
PullRequestReviewEvent

push eventbrimsec/brim

James Kerr

commit sha 7046367a246441853ac41fc68ca36717f684d174

Add tests and refactor search functions (#1049) No longer store last ts of search, use the Last.getSearch function instead Consolidate common test setup Convert pins to functional component We want to save the current program, not previous Tests for submitSearch refactored executeSearch Refactored to use issue and emit Moved common search things to a search module Remove old search object

view details

push time in 17 days

delete branch brimsec/brim

delete branch : submit-search-tests

delete time in 17 days

PR merged brimsec/brim

Submit Search Tests and Refactor

Now that we have a properly tested zealot client and an easy way to stub responses, I wrote a series of tests to lock in the behavior we expect from a user hitting "enter" in several scenarios. Once the tests were in place, I pulled out common procedures into their own functions. I'm more pleased with the API for searching within the app now. No new behavior has been added to the app here.

The parts of the app that rely on this code are:

  • Issuing a search for events
  • Issue a search for analytics
  • Issuing a search for index
  • Scrolling down the the bottom of the viewer to issue a "fetch next page" search
  • Killing a search via the three dot menu
  • Populating the uid correlation in the detail pane
  • Populating the hash correlation in the detail pane for a files log
  • The auto-refresh behavior during ingest
  • Clicking on a history entry in the tree and linear view

Example

To issue a search within brim, use the search() function defined in src/js/flows/search/mod.js.

const {response, promise, abort} = search({
  query: "_path=conn"
  from: new Date(123),
  to: new Date(456),
  spaceId: "sp_789,
  id: "Viewer",
  target: "events"
})

The response can be used to setup callbacks for each ndjson payload in the response stream.

response
  .status(status => setStatus(status))
  .chan(0, (records) => setRecords(records))
  .error(e => handleError(e))

The promise is resolved when the request finishes successfully and rejects if there is an error.

promise
  .then(handleSuccess)
  .catch(handleError)
  .finally(cleanup)

The abort function is self-explanitory. It is useful if you are issuing a search within a component and need to provide a cleanup function.

useEffect(() => {
  const {abort, response} = search(args)
  // handle response
  return () => abort() 
})

The Diff

The diff may seem intimidating, but only diffing the changes to source reveals about 600 lines modified. The rest are tests and test snapshots.

$ git diff master...head --stat ':!*test*' ':!*txt'

45 files changed, 536 insertions(+), 596 deletions(-)

 src/js/brim/index.js                                       |   3 -
 src/js/components/FilterTree.js                            |   2 +-
 src/js/components/IngestRefresh.js                         |   4 +-
 src/js/components/Investigation/FindingCard.js             |   2 +-
 src/js/components/LogDetails/Md5Panel.js                   |  17 ++---
 src/js/components/SearchBar/Input.js                       |   2 +-
 src/js/components/SearchBar/Pins.js                        |  68 ++++++------------
 src/js/components/SearchBar/SubmitButton.js                |   2 +-
 src/js/components/SearchBar/mod.js                         |   4 +-
 src/js/components/Span/SpanControls.js                     |   5 +-
 src/js/components/UidTimeline.js                           |   2 +-
 src/js/components/Viewer/HeaderCell.js                     |   2 +-
 src/js/components/charts/MainHistogram/useMainHistogram.js |   2 +-
 src/js/components/useSpanPickerMenu.js                     |   2 +-
 src/js/electron/menu/actions/searchActions.js              |  14 ++--
 src/js/flows/executeHistogramSearch.js                     |  43 -----------
 src/js/flows/executeSearch.js                              |  94 ------------------------
 src/js/flows/executeTableSearch.js                         |  69 ------------------
 src/js/flows/executeUidSearch.js                           |  41 -----------
 src/js/flows/fetchNextPage.js                              |  14 +++-
 src/js/flows/initSpace.js                                  |   4 +-
 src/js/flows/search/handler.js                             |  83 +++++++++++++++++++++
 src/js/flows/search/mod.js                                 |  39 ++++++++++
 src/js/{brim/search.js => flows/search/response.js}        |  32 +++------
 src/js/flows/searches/histogramSearch.js                   |  67 +++++++++++++++++
 src/js/flows/searches/md5Search.js                         |  28 ++++++++
 src/js/flows/searches/uidSearch.js                         |  44 ++++++++++++
 src/js/flows/searches/viewerSearch.js                      |  87 ++++++++++++++++++++++
 src/js/flows/submitAutoRefreshSearch.js                    |  32 ++++-----
 src/js/flows/submitEventsSearch.js                         |  54 --------------
 src/js/flows/submitIndexSearch.js                          | 124 --------------------------------
 src/js/flows/submitSearch.js                               |  22 ------
 src/js/flows/submitSearch/mod.js                           |  53 ++++++++++++++
 src/js/flows/submitSearch/responses/mod.js                 |   8 +++
 src/js/flows/submitSearch/save.js                          |  24 +++++++
 src/js/flows/viewLogDetail.js                              |   4 +-
 src/js/initializers/initNewSearchTab.js                    |   2 +-
 src/js/state/Handlers/selectors.js                         |   5 +-
 src/js/state/Search/reducer.js                             |   5 +-
 src/js/state/Search/types.js                               |   3 +-
 src/js/state/SearchBar/flows.js                            |   4 +-
 src/js/state/Tab/selectors.js                              |   1 -
 src/js/state/Viewer/actions.js                             |   2 +-
 src/js/state/Viewer/types.js                               |   2 +-
 zealot/fetcher/callbacks.ts                                |  11 ++-
+2741 -895

0 comment

69 changed files

jameskerr

pr closed time in 17 days

push eventbrimsec/brim

James Kerr

commit sha a274faee9a4dfbb6a16cae626f6a1841db457ae4

Add tests and refactor search functions No longer store last ts of search, use the Last.getSearch function instead Consolidate common test setup Convert pins to functional component We want to save the current program, not previous Tests for submitSearch refactored executeSearch Refactored to use issue and emit Moved common search things to a search module Remove old search object

view details

push time in 21 days

push eventbrimsec/brim

James Kerr

commit sha 5d43e5943863db6b0439b010cee3fd072b6781f9

Add tests and refactor search functions No longer store last ts of search, use the Last.getSearch function instead Consolidate common test setup Convert pins to functional component We want to save the current program, not previous Tests for submitSearch refactored executeSearch Refactored to use issue and emit Moved common search things to a search module Remove old search object

view details

push time in 22 days

push eventbrimsec/brim

James Kerr

commit sha 3e32a2ca8b6fdee15d258536dd7dc76ded5e3286

Remove old search object

view details

push time in 22 days

push eventbrimsec/brim

Matthew Nibecker

commit sha 2d6744241b6feb26687806f445d322c6374cba07

zqd: Prevent orphaned zqd (unix) (#1031) If platform is not windows, create posix pipe and pass read fd to zqd via the -brimfd argument. Should brim be closed with SIGKILL the fd will be closed and zqd, seeing the fd has closed will exit.

view details

James Kerr

commit sha 76f506a5eccf0c15ea17337d979b99f307b6708a

Index Search (#1024) * Restyle search header and add target for archives * Rebase on master * Add migration for search records * Create a sanitized space name

view details

James Kerr

commit sha df25397baa473ffa45baf966ea3190f17c910731

No longer store last ts of search, use the Last.getSearch function instead

view details

James Kerr

commit sha b5c7b5c425d009bfacce114817b120e3c58e70b2

Consolidate common test setup

view details

James Kerr

commit sha 2594784f5de3705ae9eb3ba41698a57b338edb59

Convert pins to functional component

view details

James Kerr

commit sha 67fb266135547d73fff73f829c7db460beeeee8b

We want to save the current program, not previous

view details

James Kerr

commit sha 1670fe43646ca6b4250c7c867041829f23f4973d

Tests for submitSearch

view details

James Kerr

commit sha f7e39b7a299ea45c840a821326a129bba4a01c48

refactored executeSearch

view details

James Kerr

commit sha ac736a6086a6d572394741a9a4c41be2a6c9a838

Refactored to use issue and emit

view details

James Kerr

commit sha 3de22969a9a0697bded37a2140b2332defc06138

Moved common search things to a search module

view details

James Kerr

commit sha 9f6be347ab13f2ca714caa1c5caeca06cef9db1c

WIP

view details

push time in 22 days

push eventbrimsec/brim

Brim Automation

commit sha d7d6099d27a1b09e86278021e6042c8d51e85c32

zq update through "Fix some zbuf.Batch reference counting bugs" by nwt This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1190, authored by @nwt, has been merged. Fix some zbuf.Batch reference counting bugs A zbuf.Batch implementation that includes memory reuse will panic on tests without this. Part of brimsec/zq#1091.

view details

Brim Automation

commit sha 414185a7c2d7dd2bb54cdd4db375866cdf17a60b

zq update through "zqd: Gracefully handle moved/delete pcap in spaces" by mattnibs This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1198, authored by @mattnibs, has been merged. zqd: Gracefully handle moved/delete pcap in spaces Currently a space that references a moved or delete pcap causes the list spaces endpoint to fail with 404. If a pcapspace references a no longer existing pcap make it so: - list spaces still works - info request on the space returns pcap_support false - searches on the space still works Closes brimsec/zq#1195

view details

Brim Automation

commit sha 08420a3f67fd6900776e2d0a6d666763aff4ffed

zq update through "Add record count to zar stat" by mattnibs This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1200, authored by @mattnibs, has been merged. Add record count to zar stat Additionally return record count in calls to the zqd archive stat endpoint. Closes brimsec/zq#1118

view details

Brim Automation

commit sha 045c3614a5bf2fb830e994d785a90f48a475e67a

zq update through "K8s obs" by marktwallace This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1173, authored by @marktwallace, has been merged. K8s obs This updates the procedures and config files for EKS deployment of zqd. It also adds tools for observability, including an example Grafana dashboard for zqd. The PR does not yet expose EKS endpoints, so kubectl port-forward must be used for testing.

view details

Brim Automation

commit sha 8853f2a4885ec0e8600dd9147ea3286ee28ed529

zq update through "Parallelize further when tail is order-insensitive" by henridf This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1194, authored by @henridf, has been merged. Parallelize further when tail is order-insensitive If the "tail" of a parallelized flowgraph doesn't care about input order (for example, if the flowgraph ends with ` | count()`), then we can still parallelize procs that modify the ordering field. closes brimsec/zq#1034

view details

Brim Automation

commit sha cc76058e5523a1ce9c6abd96f3e4b768a60acc65

zq update through "Parsimonious use of zng.Record.TypeCheck()" by henridf This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1192, authored by @henridf, has been merged. Parsimonious use of zng.Record.TypeCheck() We currently run TypeCheck() on every record read by a zio reader, which for some workloads adds significant overhead (see brimsec/zq#1181). Yet for data that we've imported and validated along the way (specifically, zar and zqd), this check is unneeded. This commit addresses the above by making the TypeCheck() configurable via a flag in detector.OpenConfig. zar import and zqd log ingest enable it. zq enables it and also exposes the knob through a new `-check` CLI option. It is disabled in other uses, such as `zar zq` or zqd queries. closes brimsec/zq#1181

view details

Brim Automation

commit sha 770042692a8d9e042aaf0306e91c42787ff1b11a

zq update through "Fix zar import -zngcheck" by henridf This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1205, authored by @henridf, has been merged. Fix zar import -zngcheck Immediately after merging brimsec/zq#1192, I realized that I had left the zar import zngcheck flag hard-coded to true.

view details

Brim Automation

commit sha de4bdeabfea10fcddc75db973b514b64337db96e

zq update through "Improve ast.Search handling in filter.NewBufferFilter" by nwt This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1188, authored by @nwt, has been merged. Improve ast.Search handling in filter.NewBufferFilter If you like string searches, you'll love this. * Branch more-boyer-moore ``` $ for i in 1 2 3; do /usr/bin/time ./dist/zq whatever wrccdc-year1.zng > /dev/null; done 83.14 real 124.91 user 4.85 sys 81.01 real 123.57 user 4.42 sys 83.29 real 124.79 user 4.64 sys ``` * Branch master ``` $ for i in 1 2 3; do /usr/bin/time ./dist/zq whatever wrccdc-year1.zng > /dev/null; done 275.38 real 374.43 user 10.35 sys 276.33 real 375.05 user 10.73 sys 274.70 real 373.71 user 10.15 sys ``` fieldNameFinder performance could probably be improved with caching. I'm looking into that. Closes brimsec/zq#1128.

view details

Brim Automation

commit sha 9a60609004efb08b8e15e982e71dc3ae1fbe9088

zq update through "Raise open files soft limit to hard limit in ZQL commands" by nwt This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1210, authored by @nwt, has been merged. Raise open files soft limit to hard limit in ZQL commands Addresses brimsec/zq#1167.

view details

Brim Automation

commit sha d08ac45022bf168227310c59407058706a42dfb5

zq update through "dont use error formatting for single strings" by alfred-landrum This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1199, authored by @alfred-landrum, has been merged. dont use error formatting for single strings

view details

James Kerr

commit sha 558da36b8faab936cbe185701bf456de374d3c70

Restyle search header and add target for archives

view details

James Kerr

commit sha 077e7d67b81a56636a82f838076a3549cb115d7e

Rebase on master

view details

James Kerr

commit sha c81ad7d386f8eaf766e32b79c283b944b5ddf5f3

Add migration for search records

view details

James Kerr

commit sha 957fa918b261793556e56623395f0ca3f9aea611

No longer store last ts of search, use the Last.getSearch function instead

view details

James Kerr

commit sha b8480925e1f23ac69d93cc60b0e7e09d1985e81c

Consolidate common test setup

view details

James Kerr

commit sha 05935bbbc27237fbaa0dcd7be14bc8a8c554cdda

Convert pins to functional component

view details

James Kerr

commit sha 87e39e331dd64a1f5ad349077cbfeafc8d9e7699

We want to save the current program, not previous

view details

James Kerr

commit sha 70e6cc0c01baf523b1ac683bb3d6ea690523d15f

Tests for submitSearch

view details

James Kerr

commit sha ba1af75d9d57b7e35669ad9b3d1705e1469116b2

refactored executeSearch

view details

James Kerr

commit sha 9fd26c4c2868ec3b706bbd441ef6f6ce9bf7e51a

Refactored to use issue and emit

view details

push time in 22 days

push eventbrimsec/brim

James Kerr

commit sha 76f506a5eccf0c15ea17337d979b99f307b6708a

Index Search (#1024) * Restyle search header and add target for archives * Rebase on master * Add migration for search records * Create a sanitized space name

view details

push time in 22 days

delete branch brimsec/brim

delete branch : search-header

delete time in 22 days

PR merged brimsec/brim

Index Search

Fixes #926

This PR allows searching an archive's index from within brim. After opening an archive, select "Index" from the search target selector on the left of the search bar. After issuing a valid search and getting some results, you'll be able to select one or more "chunks". Click the new "Subspace" button in the toolbar to create a new subspace based on that chunk.

Some notable changes in the code:

  • Moved components related to the SearchBar and the Toolbar into their own folders with src/js/components
  • Converted some stylesheet+className into styled-components
  • Created a new reducer to store that "last" search record. We could not rely on the history because there are valid cases when we need to issue a search without saving it to the history. However, there are many places where we need to know what the actual last search was in order to render or not render certain components.
  • Added three different icons for a space (archive, space, subspace)

image

image

image

Forgot that I need to write a migration to add the search target state.

  • [x] Write the necessary migrations
+1554 -1066

1 comment

125 changed files

jameskerr

pr closed time in 22 days

issue closedbrimsec/brim

design subspace creation from search tabular results

From an index search, we want a user to be able to choose one or more records from the tabular search results, and create a subspace where they can search the full data. We've discussed allowing control-clicking to choose records, or making checkboxes available, but haven't narrowed down to a specific design yet.

closed time in 22 days

alfred-landrum

push eventbrimsec/brim

James Kerr

commit sha cd0d608e7d60246171b8d9bea6b5ccdec4386fe2

Create a sanitized space name

view details

push time in 22 days

pull request commentbrimsec/brim

Render an error message if the initialize fails

After talking about to everyone at the Desktop Jam, we decided to not make a call to zqd until we've rendered the app. This message will be reserved for completely unexpected failures during startup.

jameskerr

comment created time in 22 days

PR closed brimsec/brim

Extract contextmenu builder

This PR just hoists some of the data needed to build the right click menu on the cells of the table into a component higher in the tree. This way, depending on what type of results are rendered, we can use different kinds of right click menus.

+22 -15

2 comments

2 changed files

jameskerr

pr closed time in 22 days

pull request commentbrimsec/brim

Extract contextmenu builder

Closing this for now. I'll revisit this PR when we actually need two different right click menus.

jameskerr

comment created time in 22 days

PullRequestReviewEvent

push eventbrimsec/brim

Phil Rzewski

commit sha 63f2f64a83d8f6f92a98e5ca90af23f194c28484

zq update through "add warnings channel to pcap-ng reader and report through zqd api (#1178)" (#1032)

view details

Brim Automation

commit sha d8b8f1edf67e5a7e970732fccb773ad3de4cc028

zq update through "zqd: prevent being orphaned by Brim (unix only)" by mattnibs This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1184, authored by @mattnibs, has been merged. zqd: prevent being orphaned by Brim (unix only) Add hidden -brimfd flag to zqd listen. Brim will use this option to prevent zqd from being orphaned in the event that Brim is terminated with a SIGKILL signal. If set zqd with listen to the file descriptor and gracefully should it be closed. PART OF brimsec/brim#1018

view details

Brim Automation

commit sha 7afffdd445499734c7e9fc28ddc34b361c8e84c5

zq update through "driver.parallelizeFlowgraph: put head/tail before and after merge" by henridf This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1182, authored by @henridf, has been merged. driver.parallelizeFlowgraph: put head/tail before and after merge `x | tail` was previously parallelized into `(x; x; ...) | tail`, which meant that all records had to be pulled through the ordered merge. It can instead be parallelized into `(x | tail; x | tail; ...) | tail` so that only the tails go through the ordered merge. The same is done for `head` for consistency, though the performance implications there are probably not significant. This change takes `zar zq -t "* | tail"` from 98 seconds to 22 seconds on my laptop for an archive with ~82m conn records. (the speedup factor is further increased when removing the TypeCheck() call per brimsec/zq#1181). This is part of brimsec/zq#1034 (and came up while looking at brimsec/zq#1172).

view details

Phil Rzewski

commit sha 4e23510c38a0a668f99550b636d821733d3f760e

Point to Zeek v3.2.0-dev-brim8 (#1033)

view details

Phil Rzewski

commit sha abfbda551d5a07b5c4bb0dc634c7673925790a32

Update zq to v0.16.0 and prep v0.20.0 (#1034) https://github.com/brimsec/brim/compare/v0.15.1...4e23510

view details

Brim Automation

commit sha 49e553ba017da1aec091d69100144a788cf90cdf

zq update through "use fewer api calls in s3 reader" by alfred-landrum This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1191, authored by @alfred-landrum, has been merged. use fewer api calls in s3 reader Use far fewer api calls to service Reads for an s3io.Reader by using a single GetObject request when the file is first read or after a Seek changes the offset.

view details

Brim Automation

commit sha e988f57c019c93769a1ac557617bf62e737890ee

zq update through "improve done and context handling in merge" by alfred-landrum This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1193, authored by @alfred-landrum, has been merged. improve done and context handling in merge Ensure that a merge runner won't call Pull on its parent after receiving a Done call, and use context cancellation to reap runner goroutines.

view details

Brim Automation

commit sha d7d6099d27a1b09e86278021e6042c8d51e85c32

zq update through "Fix some zbuf.Batch reference counting bugs" by nwt This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1190, authored by @nwt, has been merged. Fix some zbuf.Batch reference counting bugs A zbuf.Batch implementation that includes memory reuse will panic on tests without this. Part of brimsec/zq#1091.

view details

Brim Automation

commit sha 414185a7c2d7dd2bb54cdd4db375866cdf17a60b

zq update through "zqd: Gracefully handle moved/delete pcap in spaces" by mattnibs This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1198, authored by @mattnibs, has been merged. zqd: Gracefully handle moved/delete pcap in spaces Currently a space that references a moved or delete pcap causes the list spaces endpoint to fail with 404. If a pcapspace references a no longer existing pcap make it so: - list spaces still works - info request on the space returns pcap_support false - searches on the space still works Closes brimsec/zq#1195

view details

Brim Automation

commit sha 08420a3f67fd6900776e2d0a6d666763aff4ffed

zq update through "Add record count to zar stat" by mattnibs This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1200, authored by @mattnibs, has been merged. Add record count to zar stat Additionally return record count in calls to the zqd archive stat endpoint. Closes brimsec/zq#1118

view details

Brim Automation

commit sha 045c3614a5bf2fb830e994d785a90f48a475e67a

zq update through "K8s obs" by marktwallace This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1173, authored by @marktwallace, has been merged. K8s obs This updates the procedures and config files for EKS deployment of zqd. It also adds tools for observability, including an example Grafana dashboard for zqd. The PR does not yet expose EKS endpoints, so kubectl port-forward must be used for testing.

view details

Brim Automation

commit sha 8853f2a4885ec0e8600dd9147ea3286ee28ed529

zq update through "Parallelize further when tail is order-insensitive" by henridf This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1194, authored by @henridf, has been merged. Parallelize further when tail is order-insensitive If the "tail" of a parallelized flowgraph doesn't care about input order (for example, if the flowgraph ends with ` | count()`), then we can still parallelize procs that modify the ordering field. closes brimsec/zq#1034

view details

Brim Automation

commit sha cc76058e5523a1ce9c6abd96f3e4b768a60acc65

zq update through "Parsimonious use of zng.Record.TypeCheck()" by henridf This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1192, authored by @henridf, has been merged. Parsimonious use of zng.Record.TypeCheck() We currently run TypeCheck() on every record read by a zio reader, which for some workloads adds significant overhead (see brimsec/zq#1181). Yet for data that we've imported and validated along the way (specifically, zar and zqd), this check is unneeded. This commit addresses the above by making the TypeCheck() configurable via a flag in detector.OpenConfig. zar import and zqd log ingest enable it. zq enables it and also exposes the knob through a new `-check` CLI option. It is disabled in other uses, such as `zar zq` or zqd queries. closes brimsec/zq#1181

view details

Brim Automation

commit sha 770042692a8d9e042aaf0306e91c42787ff1b11a

zq update through "Fix zar import -zngcheck" by henridf This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1205, authored by @henridf, has been merged. Fix zar import -zngcheck Immediately after merging brimsec/zq#1192, I realized that I had left the zar import zngcheck flag hard-coded to true.

view details

Brim Automation

commit sha de4bdeabfea10fcddc75db973b514b64337db96e

zq update through "Improve ast.Search handling in filter.NewBufferFilter" by nwt This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1188, authored by @nwt, has been merged. Improve ast.Search handling in filter.NewBufferFilter If you like string searches, you'll love this. * Branch more-boyer-moore ``` $ for i in 1 2 3; do /usr/bin/time ./dist/zq whatever wrccdc-year1.zng > /dev/null; done 83.14 real 124.91 user 4.85 sys 81.01 real 123.57 user 4.42 sys 83.29 real 124.79 user 4.64 sys ``` * Branch master ``` $ for i in 1 2 3; do /usr/bin/time ./dist/zq whatever wrccdc-year1.zng > /dev/null; done 275.38 real 374.43 user 10.35 sys 276.33 real 375.05 user 10.73 sys 274.70 real 373.71 user 10.15 sys ``` fieldNameFinder performance could probably be improved with caching. I'm looking into that. Closes brimsec/zq#1128.

view details

Brim Automation

commit sha 9a60609004efb08b8e15e982e71dc3ae1fbe9088

zq update through "Raise open files soft limit to hard limit in ZQL commands" by nwt This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1210, authored by @nwt, has been merged. Raise open files soft limit to hard limit in ZQL commands Addresses brimsec/zq#1167.

view details

Brim Automation

commit sha d08ac45022bf168227310c59407058706a42dfb5

zq update through "dont use error formatting for single strings" by alfred-landrum This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1199, authored by @alfred-landrum, has been merged. dont use error formatting for single strings

view details

James Kerr

commit sha dc375f7831221ebdfc96a94155edbd7dafbbd649

Restyle search header and add target for archives

view details

James Kerr

commit sha 6801e26962781192d461adf55b95f97da8e8c835

Rebase on master

view details

James Kerr

commit sha 46bed5f8a1d7cf02ed0aee84710c5cd79f2f9889

Add migration for search records

view details

push time in 22 days

PR opened brimsec/brim

Submit Search Tests and Refactor

Now that we have a properly tested zealot client, and an easy way to stub responses, I wrote a series of tests to lock in the behavior we expect from a user hitting "enter" in several scenarios. Once the tests were in place, I set out to pull out common procedures into their own functions. I'm more pleased with the API for searching within the app now.

No new behavior has been added to the app here. Only a more understandable codebase.

+4152 -1846

0 comment

174 changed files

pr created time in 22 days

push eventbrimsec/brim

Phil Rzewski

commit sha 63f2f64a83d8f6f92a98e5ca90af23f194c28484

zq update through "add warnings channel to pcap-ng reader and report through zqd api (#1178)" (#1032)

view details

Brim Automation

commit sha d8b8f1edf67e5a7e970732fccb773ad3de4cc028

zq update through "zqd: prevent being orphaned by Brim (unix only)" by mattnibs This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1184, authored by @mattnibs, has been merged. zqd: prevent being orphaned by Brim (unix only) Add hidden -brimfd flag to zqd listen. Brim will use this option to prevent zqd from being orphaned in the event that Brim is terminated with a SIGKILL signal. If set zqd with listen to the file descriptor and gracefully should it be closed. PART OF brimsec/brim#1018

view details

Brim Automation

commit sha 7afffdd445499734c7e9fc28ddc34b361c8e84c5

zq update through "driver.parallelizeFlowgraph: put head/tail before and after merge" by henridf This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1182, authored by @henridf, has been merged. driver.parallelizeFlowgraph: put head/tail before and after merge `x | tail` was previously parallelized into `(x; x; ...) | tail`, which meant that all records had to be pulled through the ordered merge. It can instead be parallelized into `(x | tail; x | tail; ...) | tail` so that only the tails go through the ordered merge. The same is done for `head` for consistency, though the performance implications there are probably not significant. This change takes `zar zq -t "* | tail"` from 98 seconds to 22 seconds on my laptop for an archive with ~82m conn records. (the speedup factor is further increased when removing the TypeCheck() call per brimsec/zq#1181). This is part of brimsec/zq#1034 (and came up while looking at brimsec/zq#1172).

view details

Phil Rzewski

commit sha 4e23510c38a0a668f99550b636d821733d3f760e

Point to Zeek v3.2.0-dev-brim8 (#1033)

view details

Phil Rzewski

commit sha abfbda551d5a07b5c4bb0dc634c7673925790a32

Update zq to v0.16.0 and prep v0.20.0 (#1034) https://github.com/brimsec/brim/compare/v0.15.1...4e23510

view details

Brim Automation

commit sha 49e553ba017da1aec091d69100144a788cf90cdf

zq update through "use fewer api calls in s3 reader" by alfred-landrum This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1191, authored by @alfred-landrum, has been merged. use fewer api calls in s3 reader Use far fewer api calls to service Reads for an s3io.Reader by using a single GetObject request when the file is first read or after a Seek changes the offset.

view details

Brim Automation

commit sha e988f57c019c93769a1ac557617bf62e737890ee

zq update through "improve done and context handling in merge" by alfred-landrum This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1193, authored by @alfred-landrum, has been merged. improve done and context handling in merge Ensure that a merge runner won't call Pull on its parent after receiving a Done call, and use context cancellation to reap runner goroutines.

view details

James Kerr

commit sha ec8b19f592a2cee82e84b45b0c2c69a7b0aba72e

Restyle search header and add target for archives

view details

James Kerr

commit sha ad6b01cada8576c6a286ef026628b2833329e83a

Rebase on master

view details

James Kerr

commit sha 42d0c9efce4d29cfdaecb6016fb02eb298a28c23

Add migration for search records

view details

James Kerr

commit sha 39e35c2b8da17334a0550c4ec747b4e4b01a49a2

No longer store last ts of search, use the Last.getSearch function instead

view details

James Kerr

commit sha 6f9e7625e38bc59d7be64d847791ebb892da0d88

Consolidate common test setup

view details

James Kerr

commit sha 673b95d635ded3efc3b150612a472f6faa7a99a1

Convert pins to functional component

view details

James Kerr

commit sha 38734f4f12d0ffb72424622334b24a7782ad003d

We want to save the current program, not previous

view details

James Kerr

commit sha a82adbe3d37f197bf00c35c84a0d9f212a8823b8

Tests for submitSearch

view details

James Kerr

commit sha ff1d7918be10111154c3c40a0af07b902600023a

refactored executeSearch

view details

James Kerr

commit sha 28c8265b0bbe5e53e3c34f500572f6bbb66c5aa5

Refactored to use issue and emit

view details

James Kerr

commit sha d520838da4a0bb553444defd36717da019c134a4

Moved common search things to a search module

view details

James Kerr

commit sha fe4f6026bdab673f14a91927a84a03233e22ea19

WIP

view details

push time in 23 days

issue openedbrimsec/brim

Crash when no current space and network error

The app will crash if zqd goes down, and there is no active space.

To reproduce:

  1. Start brim
  2. pkill zqd
  3. Issue a search (to see the network error message)
  4. Open a new tab
  5. Crash with this error message

image

created time in 23 days

issue openedbrimsec/brim

Migrate to Typescript

I recommend migrating the whole codebase in one PR. Trying to make flow and typescript work together in the same project seemed like the same amount of work as doing it all at once. It might make sense for bigger projects, but we have about 30K lines of javascript that need to be ported over.

Strategy I think we should follow the instructions given in this recent blog post. The author has ported several flow projects to typescript. He says a 30K line project he recently did took him a few days.

Babel Typescript Since we already use babel to transform files in the src to the dist directory, let use the babel-typescript plugin instead of ditching babel completely for tsc. This means that we will still use tsc to type check, but the code will be built using babel. The workflow will be very similar to what we have now.

created time in 25 days

PR opened brimsec/brim

Render an error message if the initialize fails

image

+79 -20

0 comment

3 changed files

pr created time in a month

create barnchbrimsec/brim

branch : startup_error

created branch time in a month

Pull request review commentbrimsec/brim

zqd: Prevent orphaned zqd (unix)

 export class ZQD {       "-zeekrunner",       zeekRunnerCommand(this.zeekRunner)     ]++    // For unix systems, pass posix pipe read file descriptor into zqd process.+    // In the event of Brim getting shutdown via `SIGKILL`, this will let zqd+    // know that it has been orphaned and to shutdown.+    if (process.platform !== "win32") {+      const {readfd} = require("node-pipe").pipeSync()+      opts.stdio.push(readfd)+      args.push(`-brimfd=${opts.stdio.length - 1}`)

I'm not sure how this works so I may be off, but it looks like you'll pass in the integer 3 here. Were you intending to access the array opts.stdio[opts.stdio.length - 1]?

mattnibs

comment created time in a month

PullRequestReviewEvent

create barnchbrimsec/brim

branch : submit-search-tests

created branch time in a month

push eventbrimsec/brim

Brim Automation

commit sha a128281c84ae651328fce646d04c3c851d5b9eac

zq update through "Clean up filter/filter_test.go" by nwt This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1177, authored by @nwt, has been merged. Clean up filter/filter_test.go

view details

Brim Automation

commit sha 01953f42661b7edf76b581250f7e53ce38917578

zq update through "Fix bug in AST parallelization with cut/put/rename" by henridf This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1171, authored by @henridf, has been merged. Fix bug in AST parallelization with cut/put/rename Fix issue spotted by @nwt here: https://github.com/brimsec/zq/pull/1145/files/e65c62a62b9cf59e5acff3cde111b23c0029fe3b#r475911797

view details

Brim Automation

commit sha ee5d3fe027a73eb96a7728c5474923ddb6e876ec

zq update through "Enable parallel flowgraph" by henridf This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1165, authored by @henridf, has been merged. Enable parallel flowgraph Putting this up as a draft as I still need to add some tests. closes brimsec/zq#1155

view details

Brim Automation

commit sha 8de01e8709b3d39b21f133ef83ef267e1d3e1e88

zq update through "Add Done() handling to ordered merge" by henridf This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1180, authored by @henridf, has been merged. Add Done() handling to ordered merge closes brimsec/zq#1170

view details

Remy

commit sha 84dcd3ea86f88a1617e366d0402ae4d3679b2295

Column chooser sorting and searching (#1012) * Sorting is working. * Filtering is working. * Button placement is working. Hardcoded though. Implements brimsec#949 and brimsec#975 * Fixed formatting error. * Fixes brimsec#982 * Made changes @jameskerr suggested. * Update src/js/components/ColumnChooserMenu.js Co-authored-by: James Kerr <jkerr838@gmail.com> * Update src/css/_column-chooser.scss Co-authored-by: James Kerr <jkerr838@gmail.com> * Format with prettier Co-authored-by: James Kerr <jkerr838@gmail.com>

view details

Phil Rzewski

commit sha d7beea8a22e344d1b076c4a20bc6f23650fbae1f

Additional debug steps for pcap extraction problems (#1020)

view details

Brim Automation

commit sha 7cf09ff55ad9332810eaf607a06f6eb303e3cf61

zq update through "remove an unncessary waitgroup in log ingester" by mccanne This is an auto-generated commit with a zq dependency update. The zq PR https://github.com/brimsec/zq/pull/1176, authored by @mccanne, has been merged. remove an unncessary waitgroup in log ingester This commit removes an unncessary waitgroup from the log ingester since synchronous was already being provided by the closure of the warnings channel. We also added comments for clarification. This follows more closely the pattern of context.Context.

view details

James Kerr

commit sha b6687e0b96cf4b033bf9b9ebfe8ac2c73e17f345

Fix bug where span was mutating (#1027) When fetching the next page, the time span was being mutated

view details

Mason Fish

commit sha 8620029f6c5796b6dafb48b0deda8c564c10e98b

New Connection page (#1007) * design ui Signed-off-by: Mason Fish <mason@looky.cloud> * wire up functionality Signed-off-by: Mason Fish <mason@looky.cloud> * add status check Signed-off-by: Mason Fish <mason@looky.cloud> * cleanup Signed-off-by: Mason Fish <mason@looky.cloud> * pass in createZealot, fix bugs, code complete Signed-off-by: Mason Fish <mason@looky.cloud> * fix current tests Signed-off-by: Mason Fish <mason@looky.cloud> * add migration Signed-off-by: Mason Fish <mason@looky.cloud> * add flow test Signed-off-by: Mason Fish <mason@looky.cloud> * rearrange to modal Signed-off-by: Mason Fish <mason@looky.cloud> * fix log detail window for new connection Signed-off-by: Mason Fish <mason@looky.cloud> * don't close modal on error Signed-off-by: Mason Fish <mason@looky.cloud> * address comments Signed-off-by: Mason Fish <mason@looky.cloud> Co-authored-by: Mason Fish <mason@looky.cloud>

view details

Phil Rzewski

commit sha cb82aef56b42911e2674690df4050524d453a303

ZQL docs links & better link for opening issues (#1030)

view details

James Kerr

commit sha 6cdc25ecd660bcc3b23dddac684eba9e7884ddee

Restyle search header and add target for archives

view details

James Kerr

commit sha 618e3183867ae3935a9b6e1655dd52731a9d20cf

Rebase on master

view details

James Kerr

commit sha 3966a8e364ee44a24b5e3c2ec2bc558501af338f

Add migration for search records

view details

push time in a month

issue commentbrimsec/brim

Connection Detail view

I'll gladly work on some designs for this.

mason-fish

comment created time in a month

push eventbrimsec/brim

James Kerr

commit sha b6687e0b96cf4b033bf9b9ebfe8ac2c73e17f345

Fix bug where span was mutating (#1027) When fetching the next page, the time span was being mutated

view details

push time in a month

delete branch brimsec/brim

delete branch : packet-button

delete time in a month

PR merged brimsec/brim

Fix bug where span was mutating

When fetching the next page, the time span was being mutated causing weird bugs like the packet button not lighting up when clicking on a record. This also explains why it was working fine in the detail view. The span was not mutated there I suppose.

Fixes #979

+13 -5

1 comment

2 changed files

jameskerr

pr closed time in a month

issue closedbrimsec/brim

"Packets" button not activating after lots of infinite scrolling

Reproducing this issue is unfortunately somewhat complicated, but I think I've come up with a fairly reliable recipe at least. Repro is on Brim commit 7c8527a talking to zqd commit a1a817e.

  1. Start with a pristine instance of the app (i.e. appState.json and Spaces directories removed before launching).

  2. Import an uncompressed copy of wrccdc.2018-03-23.010014000000000.pcap.gz and let generation of Zeek logs completely finish.

  3. Drag down the vertical scrollbar to trigger infinite scroll. I keep wiggling the mouse pointer up & down near the bottom of the scrollbar, which seems to help re-triggering the infinite scroll. I keep doing this for over 30 seconds and use a stopwatch to make sure I've waited at least that long.

  4. Single-click on one of the conn records in view. At this point I consistently find the "Packets" button does not activate, giving me the impression as a user that no flow is available in the original pcap corresponding to this conn record.

  5. Double-click the same conn record. In the separate Log Detail window that opens, I find the "Packets" button is now activated.

Steps 3-5 are shown in the attached video. It's not shown in the video, but clicking the "Packets" button does bring up the flow successfully in Wireshark.

Repro.mp4.zip

Note that due to the timing issues with triggering infinite scroll, it would be nearly impossible to click the exact same flow in each repro. However, I've reproduced this several times using the same recipe and found that clicking any conn record that far down in the scrolling reproduces the issue.

It also seems that the infinite scrolling may somehow be important to triggering the issue. For example, after reproducing the behavior with a particular flow, I've created ZQL to isolate the same flow via 4-tuple and executed that ZQL in a fresh version of the app after re-importing the same PCAP, but in that case I've been ale to reliably activate the "Packets" button on the single conn record that comes up in that case.

Finally, I had a conversation on Slack with @jameskerr about what triggers the "Packets" button to activate. His explanation:

We take the uid from the log if there is one, then we execute this query and see if there is a conn event.

uid=${uid} or ${uid} in conn_uids or ${uid} in uids or referenced_file.uid=${uid} | head ${UID_CORRELATION_LIMIT}

With the limit being 100 Then if there is a conn log in that set, and the space has “packet_support” set to true, we’ll enable the packets button.

Since this confirmed a backend query was involved, I used Wireshark to isolate the backend query in the non-working and working cases. Here's what comes up in the TCP byte stream (includes both directions of the conversation):

Button not activating

POST /search?format=zjson HTTP/1.1
Host: localhost:9867
Connection: keep-alive
Content-Length: 1019
sec-ch-ua: 
Origin: file://
Sec-Fetch-Dest: empty
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Brim/0.14.0 Chrome/78.0.3904.130 Electron/7.3.2 Safari/537.36
Content-Type: text/plain;charset=UTF-8
Accept: */*
Sec-Fetch-Site: cross-site
Sec-Fetch-Mode: cors
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US

{"proc":{"op":"SequentialProc","procs":[{"op":"FilterProc","filter":{"op":"LogicalOr","left":{"op":"LogicalOr","left":{"op":"LogicalOr","left":{"op":"CompareField","comparator":"=","field":{"op":"FieldRead","field":"uid"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}},"right":{"op":"CompareField","comparator":"in","field":{"op":"FieldRead","field":"conn_uids"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}}},"right":{"op":"CompareField","comparator":"in","field":{"op":"FieldRead","field":"uids"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}}},"right":{"op":"CompareField","comparator":"=","field":{"op":"FieldCall","fn":"RecordFieldRead","field":{"op":"FieldRead","field":"referenced_file"},"param":"uid"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}}}},{"op":"HeadProc","count":100}]},"span":{"ts":{"sec":1521835102,"ns":636000000},"dur":{"sec":111,"ns":660000000}},"space":"sp_1fvlMFN03xJzNXhmbo4S729sBEp","dir":-1}HTTP/1.1 200 OK
Content-Type: application/x-ndjson
X-Request-Id: 47
Date: Tue, 11 Aug 2020 00:45:30 GMT
Transfer-Encoding: chunked

23
{"type":"TaskStart","task_id":0}



35
{"type":"SearchEnd","channel_id":0,"reason":"eof"}



ca
{"type":"SearchStats","start_time":{"sec":1597106730,"ns":464115000},"update_time":{"sec":1597106730,"ns":921383000},"bytes_read":38147100,"bytes_matched":0,"records_read":407000,"records_matched":0}



1f
{"type":"TaskEnd","task_id":0}

0

Button activating when opening Log Detail window

POST /search?format=zjson HTTP/1.1
Host: localhost:9867
Connection: keep-alive
Content-Length: 1019
Origin: file://
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Brim/0.14.0 Chrome/78.0.3904.130 Electron/7.3.2 Safari/537.36
Content-Type: text/plain;charset=UTF-8
Accept: */*
Sec-Fetch-Site: cross-site
Sec-Fetch-Mode: cors
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US

{"proc":{"op":"SequentialProc","procs":[{"op":"FilterProc","filter":{"op":"LogicalOr","left":{"op":"LogicalOr","left":{"op":"LogicalOr","left":{"op":"CompareField","comparator":"=","field":{"op":"FieldRead","field":"uid"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}},"right":{"op":"CompareField","comparator":"in","field":{"op":"FieldRead","field":"conn_uids"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}}},"right":{"op":"CompareField","comparator":"in","field":{"op":"FieldRead","field":"uids"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}}},"right":{"op":"CompareField","comparator":"=","field":{"op":"FieldCall","fn":"RecordFieldRead","field":{"op":"FieldRead","field":"referenced_file"},"param":"uid"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}}}},{"op":"HeadProc","count":100}]},"span":{"ts":{"sec":1521835102,"ns":636000000},"dur":{"sec":111,"ns":696000000}},"space":"sp_1fvlMFN03xJzNXhmbo4S729sBEp","dir":-1}HTTP/1.1 200 OK
Content-Type: application/x-ndjson
X-Request-Id: 51
Date: Tue, 11 Aug 2020 00:46:06 GMT
Transfer-Encoding: chunked

23
{"type":"TaskStart","task_id":0}



cb
{"type":"SearchStats","start_time":{"sec":1597106766,"ns":998738000},"update_time":{"sec":1597106767,"ns":504964000},"bytes_read":35693265,"bytes_matched":89,"records_read":390117,"records_matched":1}



649
{"type":"SearchRecords","channel_id":0,"records":[{"id":29,"type":[{"name":"_path","type":"string"},{"name":"ts","type":"time"},{"name":"uid","type":"bstring"},{"name":"id","type":[{"name":"orig_h","type":"ip"},{"name":"orig_p","type":"port"},{"name":"resp_h","type":"ip"},{"name":"resp_p","type":"port"}]},{"name":"proto","type":"zenum"},{"name":"service","type":"bstring"},{"name":"duration","type":"duration"},{"name":"orig_bytes","type":"uint64"},{"name":"resp_bytes","type":"uint64"},{"name":"conn_state","type":"bstring"},{"name":"local_orig","type":"bool"},{"name":"local_resp","type":"bool"},{"name":"missed_bytes","type":"uint64"},{"name":"history","type":"bstring"},{"name":"orig_pkts","type":"uint64"},{"name":"orig_ip_bytes","type":"uint64"},{"name":"resp_pkts","type":"uint64"},{"name":"resp_ip_bytes","type":"uint64"},{"name":"tunnel_parents","type":"set[bstring]"},{"name":"geo","type":[{"name":"orig","type":[{"name":"country_code","type":"bstring"},{"name":"region","type":"bstring"},{"name":"city","type":"bstring"},{"name":"latitude","type":"float64"},{"name":"longitude","type":"float64"}]},{"name":"resp","type":[{"name":"country_code","type":"bstring"},{"name":"region","type":"bstring"},{"name":"city","type":"bstring"},{"name":"latitude","type":"float64"},{"name":"longitude","type":"float64"}]}]}],"aliases":[{"name":"zenum","type":"string"}],"values":["conn","1521835214.297815","C52cuf1JdFLKEYXdv3",["10.236.58.83","47941","10.47.1.50","15765"],"tcp",null,"0.000034","0","0","S0",null,null,"0","S","2","80","0","0",null,[[null,null,null,null,null],[null,null,null,null,null]]]}]}



35
{"type":"SearchEnd","channel_id":0,"reason":"eof"}



cb
{"type":"SearchStats","start_time":{"sec":1597106766,"ns":998738000},"update_time":{"sec":1597106767,"ns":528221000},"bytes_read":38191521,"bytes_matched":89,"records_read":407495,"records_matched":1}



1f
{"type":"TaskEnd","task_id":0}

0

If I isolate just the query portions, run each through jq, and diff them, I find only one difference:

$ echo '{"proc":{"op":"SequentialProc","procs":[{"op":"FilterProc","filter":{"op":"LogicalOr","left":{"op":"LogicalOr","left":{"op":"LogicalOr","left":{"op":"CompareField","comparator":"=","field":{"op":"FieldRead","field":"uid"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}},"right":{"op":"CompareField","comparator":"in","field":{"op":"FieldRead","field":"conn_uids"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}}},"right":{"op":"CompareField","comparator":"in","field":{"op":"FieldRead","field":"uids"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}}},"right":{"op":"CompareField","comparator":"=","field":{"op":"FieldCall","fn":"RecordFieldRead","field":{"op":"FieldRead","field":"referenced_file"},"param":"uid"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}}}},{"op":"HeadProc","count":100}]},"span":{"ts":{"sec":1521835102,"ns":636000000},"dur":{"sec":111,"ns":660000000}},"space":"sp_1fvlMFN03xJzNXhmbo4S729sBEp","dir":-1}' | jq . > not-working.json 

$ echo '{"proc":{"op":"SequentialProc","procs":[{"op":"FilterProc","filter":{"op":"LogicalOr","left":{"op":"LogicalOr","left":{"op":"LogicalOr","left":{"op":"CompareField","comparator":"=","field":{"op":"FieldRead","field":"uid"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}},"right":{"op":"CompareField","comparator":"in","field":{"op":"FieldRead","field":"conn_uids"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}}},"right":{"op":"CompareField","comparator":"in","field":{"op":"FieldRead","field":"uids"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}}},"right":{"op":"CompareField","comparator":"=","field":{"op":"FieldCall","fn":"RecordFieldRead","field":{"op":"FieldRead","field":"referenced_file"},"param":"uid"},"value":{"op":"Literal","type":"string","value":"C52cuf1JdFLKEYXdv3"}}}},{"op":"HeadProc","count":100}]},"span":{"ts":{"sec":1521835102,"ns":636000000},"dur":{"sec":111,"ns":696000000}},"space":"sp_1fvlMFN03xJzNXhmbo4S729sBEp","dir":-1}' | jq . > working.json

$ diff not-working.json working.json 
87c87
<       "ns": 660000000
---
>       "ns": 696000000

The slightly longer value for the duration is suspicious. I saw the same diff on multiple repros. Is it possible that there's different query generators for the main events panel and Log Detail, with a bug in the former?

closed time in a month

philrz

Pull request review commentbrimsec/brim

Fix bug where span was mutating

 test("#fetchNextPage when there is only 1 event", () => {     })   ) })++test("#fetchNextPage does not mutate the exisiting span", () => {+  const before = [...Tab.getSpanAsDates(store.getState())]+  store.dispatch(fetchNextPage())+  const after = [...Tab.getSpanAsDates(store.getState())]++  expect(before).toEqual(after)+})

Before adding the fix, this test was failing as a result of the mutation.

jameskerr

comment created time in a month

PullRequestReviewEvent

PR opened brimsec/brim

Fix bug where span was mutating

When fetching the next page, the time span was being mutated causing weird bugs like the packet button no lighting up when clicking on a conn. This also explains why it was working fine in the detail view. The span was not mutated there I suppose.

Fixes #979

+13 -5

0 comment

2 changed files

pr created time in a month

more