profile
viewpoint

Pull request review commentelastic/cloud-on-k8s

Fix GCloud auth to work on CI

 func (d *OcpDriver) auth() error { 		if err := client.ReadIntoFile(keyFileName, OcpVaultPath, OcpServiceAccountVaultFieldName); err != nil { 			return err 		}-		log.Println("Setting $GCLOUD_KEYFILE_JSON...")-		return os.Setenv("GCLOUD_KEYFILE_JSON", keyFileName)++		if d.plan.Ocp.UseNonDefaultCloudSDKPath {+			// ensure gcloud & gsutil rely on credentials stored in gcpDir instead of using the default+			// directory (~/.config/gcloud), to not tamper any default gcloud auth already set on the system+			log.Printf(fmt.Sprintf("Setting $CLOUDSDK_CONFIG=%s", gcpDir))+			if err := os.Setenv("CLOUDSDK_CONFIG", gcpDir); err != nil {+				return err+			}+		}+		if err := NewCommand(fmt.Sprintf("gcloud config set project %s", d.ctx["GCloudProject"])).Run(); err != nil {+			return err+		}

A-ha, fair!

Would you mind adding a comment explaining why this call is here? Otherwise, someone may try to deduplicate this by doing what I suggested without knowing the consequences. :smile:

sebgl

comment created time in a month

Pull request review commentelastic/cloud-on-k8s

Fix GCloud auth to work on CI

 func (d *OcpDriver) auth() error { 		if err := client.ReadIntoFile(keyFileName, OcpVaultPath, OcpServiceAccountVaultFieldName); err != nil { 			return err 		}-		log.Println("Setting $GCLOUD_KEYFILE_JSON...")-		return os.Setenv("GCLOUD_KEYFILE_JSON", keyFileName)++		if d.plan.Ocp.UseNonDefaultCloudSDKPath {+			// ensure gcloud & gsutil rely on credentials stored in gcpDir instead of using the default+			// directory (~/.config/gcloud), to not tamper any default gcloud auth already set on the system+			log.Printf(fmt.Sprintf("Setting $CLOUDSDK_CONFIG=%s", gcpDir))+			if err := os.Setenv("CLOUDSDK_CONFIG", gcpDir); err != nil {+				return err+			}+		}+		if err := NewCommand(fmt.Sprintf("gcloud config set project %s", d.ctx["GCloudProject"])).Run(); err != nil {+			return err+		}

I think we can move this to the top of the function and remove the call in L226. I believe this is how it was before the previous PR.

sebgl

comment created time in a month

pull request commentopenshift/release

ci-operator/config/openshift/installer/openshift-installer-master.yaml: Set containers_image_openpgp

@abhinavdahiya @wking @crawford o/ folks! Can one of you please remove me from the assignees?

wking

comment created time in a month

Pull request review commentelastic/cloud-on-k8s

Fix GCloud auth to work on CI

 func (d *OcpDriver) Execute() error { }  func (d *OcpDriver) auth() error {+	log.Println("Authenticating as service account...") -	if d.plan.ServiceAccount {-		log.Println("Authenticating as service account...")--		client, err := NewClient(*d.plan.VaultInfo)-		if err != nil {-			return err-		}+	client, err := NewClient(*d.plan.VaultInfo)+	if err != nil {+		return err+	} -		gcpDir := filepath.Join(os.Getenv("HOME"), ".gcp")-		keyFileName := filepath.Join(os.Getenv("HOME"), ".gcp", "osServiceAccount.json")-		_ = os.MkdirAll(gcpDir, os.ModePerm)+	gcpDir := filepath.Join(os.Getenv("HOME"), ".gcp")+	keyFileName := filepath.Join(os.Getenv("HOME"), ".gcp", "osServiceAccount.json")+	_ = os.MkdirAll(gcpDir, os.ModePerm) -		if err := client.ReadIntoFile(keyFileName, OcpVaultPath, OcpServiceAccountVaultFieldName); err != nil {-			return err-		}-		log.Println("Setting $GCLOUD_KEYFILE_JSON...")-		return os.Setenv("GCLOUD_KEYFILE_JSON", keyFileName)+	if err := client.ReadIntoFile(keyFileName, OcpVaultPath, OcpServiceAccountVaultFieldName); err != nil {+		return err 	} -	log.Println("Authenticating as user...")-	accounts, err := NewCommand(`gcloud auth list "--format=value(account)"`).StdoutOnly().WithoutStreaming().Output()-	if err != nil {+	// ensure gcloud & gsutil rely on credentials stored in gcpDir instead of using the default+	// directory (~/.config/gcloud), to not tamper any default gcloud auth already set on the system+	if err := os.Setenv("CLOUDSDK_CONFIG", gcpDir); err != nil { 		return err 	}--	if len(accounts) > 0 {-		return nil+	if err := NewCommand(fmt.Sprintf("gcloud config set project %s", d.ctx["GCloudProject"])).Run(); err != nil {+		return err 	}--	_ = NewCommand(fmt.Sprintf("gcloud config set project %s", d.ctx["GCloudProject"])).Run()-	return NewCommand("gcloud auth login").Run()

Is the login action redundant? I remember it being required regardless. Did it cause any issues for you?

sebgl

comment created time in a month

Pull request review commentelastic/cloud-on-k8s

Fix GCloud auth to work on CI

 func (d *OcpDriver) Execute() error { }  func (d *OcpDriver) auth() error {+	log.Println("Authenticating as service account...") -	if d.plan.ServiceAccount {

Removing this line makes accessing vault (and having the service account stored in vault) always required, which means everyone will be using OcpVaultPath unless it's configured/overwritten.

sebgl

comment created time in a month

Pull request review commentelastic/cloud-on-k8s

Authenticate to GCP using an env var in the ocp deployer

 func (d *OcpDriver) Execute() error { }  func (d *OcpDriver) auth() error {-	_ = NewCommand(fmt.Sprintf("gcloud config set project %s", d.ctx["GCloudProject"])).Run()

Curious why you moved this to the bottom of the function.

sebgl

comment created time in a month

issue openedelastic/logstash

Logstash fails to start when the queue size is bigger than the available memory

After some debugging and an amazing support from @andsel and @jsvd, we found out that logstash seems to be failing to start when persisted queues are bigger than the available memory.

We have a logstash deployment that is configured with the following pipeline file:

  pipelines.yml: |
    - path.config: /usr/share/logstash/pipeline/ea-input.conf
      pipeline.id: input
      pipeline.workers: 1
    - path.config: /usr/share/logstash/pipeline/riemann.conf
      pipeline.id: riemann
      pipeline.workers: 1
      queue.max_bytes: 40gb
      queue.type: persisted
    - path.config: /usr/share/logstash/pipeline/ea-elasticsearch.conf
      pipeline.id: elasticsearch
      pipeline.workers: 2
      queue.max_bytes: 40gb
      queue.type: persisted

Logstash failed to bring up the riemann queue after a restart. After debugging, we noticed that the size of the queue files is definitely bigger than the memory we have available:

bash-4.2$ ls data/queue/riemann
checkpoint.6109  checkpoint.6159  checkpoint.6209  checkpoint.6259  checkpoint.6309  checkpoint.6359  checkpoint.6409  checkpoint.6459  checkpoint.6509  checkpoint.6559  checkpoint.6609  page.6115  page.6165  page.6215  page.6265  page.6315  page.6365  page.6415  page.6465  page.6515  page.6565  page.6615
checkpoint.6110  checkpoint.6160  checkpoint.6210  checkpoint.6260  checkpoint.6310  checkpoint.6360  checkpoint.6410  checkpoint.6460  checkpoint.6510  checkpoint.6560  checkpoint.6610  page.6116  page.6166  page.6216  page.6266  page.6316  page.6366  page.6416  page.6466  page.6516  page.6566  page.6616
checkpoint.6111  checkpoint.6161  checkpoint.6211  checkpoint.6261  checkpoint.6311  checkpoint.6361  checkpoint.6411  checkpoint.6461  checkpoint.6511  checkpoint.6561  checkpoint.6611  page.6117  page.6167  page.6217  page.6267  page.6317  page.6367  page.6417  page.6467  page.6517  page.6567  page.6617
checkpoint.6112  checkpoint.6162  checkpoint.6212  checkpoint.6262  checkpoint.6312  checkpoint.6362  checkpoint.6412  checkpoint.6462  checkpoint.6512  checkpoint.6562  checkpoint.6612  page.6118  page.6168  page.6218  page.6268  page.6318  page.6368  page.6418  page.6468  page.6518  page.6568  page.6618
checkpoint.6113  checkpoint.6163  checkpoint.6213  checkpoint.6263  checkpoint.6313  checkpoint.6363  checkpoint.6413  checkpoint.6463  checkpoint.6513  checkpoint.6563  checkpoint.6613  page.6119  page.6169  page.6219  page.6269  page.6319  page.6369  page.6419  page.6469  page.6519  page.6569  page.6619
checkpoint.6114  checkpoint.6164  checkpoint.6214  checkpoint.6264  checkpoint.6314  checkpoint.6364  checkpoint.6414  checkpoint.6464  checkpoint.6514  checkpoint.6564  checkpoint.6614  page.6120  page.6170  page.6220  page.6270  page.6320  page.6370  page.6420  page.6470  page.6520  page.6570  page.6620
checkpoint.6115  checkpoint.6165  checkpoint.6215  checkpoint.6265  checkpoint.6315  checkpoint.6365  checkpoint.6415  checkpoint.6465  checkpoint.6515  checkpoint.6565  checkpoint.6615  page.6121  page.6171  page.6221  page.6271  page.6321  page.6371  page.6421  page.6471  page.6521  page.6571  page.6621
checkpoint.6116  checkpoint.6166  checkpoint.6216  checkpoint.6266  checkpoint.6316  checkpoint.6366  checkpoint.6416  checkpoint.6466  checkpoint.6516  checkpoint.6566  checkpoint.6616  page.6122  page.6172  page.6222  page.6272  page.6322  page.6372  page.6422  page.6472  page.6522  page.6572  page.6622
checkpoint.6117  checkpoint.6167  checkpoint.6217  checkpoint.6267  checkpoint.6317  checkpoint.6367  checkpoint.6417  checkpoint.6467  checkpoint.6517  checkpoint.6567  checkpoint.6617  page.6123  page.6173  page.6223  page.6273  page.6323  page.6373  page.6423  page.6473  page.6523  page.6573  page.6623
checkpoint.6118  checkpoint.6168  checkpoint.6218  checkpoint.6268  checkpoint.6318  checkpoint.6368  checkpoint.6418  checkpoint.6468  checkpoint.6518  checkpoint.6568  checkpoint.6618  page.6124  page.6174  page.6224  page.6274  page.6324  page.6374  page.6424  page.6474  page.6524  page.6574  page.6624
checkpoint.6119  checkpoint.6169  checkpoint.6219  checkpoint.6269  checkpoint.6319  checkpoint.6369  checkpoint.6419  checkpoint.6469  checkpoint.6519  checkpoint.6569  checkpoint.6619  page.6125  page.6175  page.6225  page.6275  page.6325  page.6375  page.6425  page.6475  page.6525  page.6575  page.6625
checkpoint.6120  checkpoint.6170  checkpoint.6220  checkpoint.6270  checkpoint.6320  checkpoint.6370  checkpoint.6420  checkpoint.6470  checkpoint.6520  checkpoint.6570  checkpoint.6620  page.6126  page.6176  page.6226  page.6276  page.6326  page.6376  page.6426  page.6476  page.6526  page.6576  page.6626
checkpoint.6121  checkpoint.6171  checkpoint.6221  checkpoint.6271  checkpoint.6321  checkpoint.6371  checkpoint.6421  checkpoint.6471  checkpoint.6521  checkpoint.6571  checkpoint.6621  page.6127  page.6177  page.6227  page.6277  page.6327  page.6377  page.6427  page.6477  page.6527  page.6577  page.6627
checkpoint.6122  checkpoint.6172  checkpoint.6222  checkpoint.6272  checkpoint.6322  checkpoint.6372  checkpoint.6422  checkpoint.6472  checkpoint.6522  checkpoint.6572  checkpoint.6622  page.6128  page.6178  page.6228  page.6278  page.6328  page.6378  page.6428  page.6478  page.6528  page.6578  page.6628
checkpoint.6123  checkpoint.6173  checkpoint.6223  checkpoint.6273  checkpoint.6323  checkpoint.6373  checkpoint.6423  checkpoint.6473  checkpoint.6523  checkpoint.6573  checkpoint.6623  page.6129  page.6179  page.6229  page.6279  page.6329  page.6379  page.6429  page.6479  page.6529  page.6579  page.6629
checkpoint.6124  checkpoint.6174  checkpoint.6224  checkpoint.6274  checkpoint.6324  checkpoint.6374  checkpoint.6424  checkpoint.6474  checkpoint.6524  checkpoint.6574  checkpoint.6624  page.6130  page.6180  page.6230  page.6280  page.6330  page.6380  page.6430  page.6480  page.6530  page.6580  page.6630
checkpoint.6125  checkpoint.6175  checkpoint.6225  checkpoint.6275  checkpoint.6325  checkpoint.6375  checkpoint.6425  checkpoint.6475  checkpoint.6525  checkpoint.6575  checkpoint.6625  page.6131  page.6181  page.6231  page.6281  page.6331  page.6381  page.6431  page.6481  page.6531  page.6581  page.6631
checkpoint.6126  checkpoint.6176  checkpoint.6226  checkpoint.6276  checkpoint.6326  checkpoint.6376  checkpoint.6426  checkpoint.6476  checkpoint.6526  checkpoint.6576  checkpoint.6626  page.6132  page.6182  page.6232  page.6282  page.6332  page.6382  page.6432  page.6482  page.6532  page.6582  page.6632
checkpoint.6127  checkpoint.6177  checkpoint.6227  checkpoint.6277  checkpoint.6327  checkpoint.6377  checkpoint.6427  checkpoint.6477  checkpoint.6527  checkpoint.6577  checkpoint.6627  page.6133  page.6183  page.6233  page.6283  page.6333  page.6383  page.6433  page.6483  page.6533  page.6583  page.6633
checkpoint.6128  checkpoint.6178  checkpoint.6228  checkpoint.6278  checkpoint.6328  checkpoint.6378  checkpoint.6428  checkpoint.6478  checkpoint.6528  checkpoint.6578  checkpoint.6628  page.6134  page.6184  page.6234  page.6284  page.6334  page.6384  page.6434  page.6484  page.6534  page.6584  page.6634
checkpoint.6129  checkpoint.6179  checkpoint.6229  checkpoint.6279  checkpoint.6329  checkpoint.6379  checkpoint.6429  checkpoint.6479  checkpoint.6529  checkpoint.6579  checkpoint.6629  page.6135  page.6185  page.6235  page.6285  page.6335  page.6385  page.6435  page.6485  page.6535  page.6585  page.6635
checkpoint.6130  checkpoint.6180  checkpoint.6230  checkpoint.6280  checkpoint.6330  checkpoint.6380  checkpoint.6430  checkpoint.6480  checkpoint.6530  checkpoint.6580  checkpoint.6630  page.6136  page.6186  page.6236  page.6286  page.6336  page.6386  page.6436  page.6486  page.6536  page.6586  page.6636
checkpoint.6131  checkpoint.6181  checkpoint.6231  checkpoint.6281  checkpoint.6331  checkpoint.6381  checkpoint.6431  checkpoint.6481  checkpoint.6531  checkpoint.6581  checkpoint.6631  page.6137  page.6187  page.6237  page.6287  page.6337  page.6387  page.6437  page.6487  page.6537  page.6587  page.6637
checkpoint.6132  checkpoint.6182  checkpoint.6232  checkpoint.6282  checkpoint.6332  checkpoint.6382  checkpoint.6432  checkpoint.6482  checkpoint.6532  checkpoint.6582  checkpoint.6632  page.6138  page.6188  page.6238  page.6288  page.6338  page.6388  page.6438  page.6488  page.6538  page.6588  page.6638
checkpoint.6133  checkpoint.6183  checkpoint.6233  checkpoint.6283  checkpoint.6333  checkpoint.6383  checkpoint.6433  checkpoint.6483  checkpoint.6533  checkpoint.6583  checkpoint.6633  page.6139  page.6189  page.6239  page.6289  page.6339  page.6389  page.6439  page.6489  page.6539  page.6589  page.6639
checkpoint.6134  checkpoint.6184  checkpoint.6234  checkpoint.6284  checkpoint.6334  checkpoint.6384  checkpoint.6434  checkpoint.6484  checkpoint.6534  checkpoint.6584  checkpoint.6634  page.6140  page.6190  page.6240  page.6290  page.6340  page.6390  page.6440  page.6490  page.6540  page.6590  page.6640
checkpoint.6135  checkpoint.6185  checkpoint.6235  checkpoint.6285  checkpoint.6335  checkpoint.6385  checkpoint.6435  checkpoint.6485  checkpoint.6535  checkpoint.6585  checkpoint.6635  page.6141  page.6191  page.6241  page.6291  page.6341  page.6391  page.6441  page.6491  page.6541  page.6591  page.6641
checkpoint.6136  checkpoint.6186  checkpoint.6236  checkpoint.6286  checkpoint.6336  checkpoint.6386  checkpoint.6436  checkpoint.6486  checkpoint.6536  checkpoint.6586  checkpoint.6636  page.6142  page.6192  page.6242  page.6292  page.6342  page.6392  page.6442  page.6492  page.6542  page.6592  page.6642
checkpoint.6137  checkpoint.6187  checkpoint.6237  checkpoint.6287  checkpoint.6337  checkpoint.6387  checkpoint.6437  checkpoint.6487  checkpoint.6537  checkpoint.6587  checkpoint.6637  page.6143  page.6193  page.6243  page.6293  page.6343  page.6393  page.6443  page.6493  page.6543  page.6593  page.6643
checkpoint.6138  checkpoint.6188  checkpoint.6238  checkpoint.6288  checkpoint.6338  checkpoint.6388  checkpoint.6438  checkpoint.6488  checkpoint.6538  checkpoint.6588  checkpoint.6638  page.6144  page.6194  page.6244  page.6294  page.6344  page.6394  page.6444  page.6494  page.6544  page.6594  page.6644
checkpoint.6139  checkpoint.6189  checkpoint.6239  checkpoint.6289  checkpoint.6339  checkpoint.6389  checkpoint.6439  checkpoint.6489  checkpoint.6539  checkpoint.6589  checkpoint.6639  page.6145  page.6195  page.6245  page.6295  page.6345  page.6395  page.6445  page.6495  page.6545  page.6595  page.6645
checkpoint.6140  checkpoint.6190  checkpoint.6240  checkpoint.6290  checkpoint.6340  checkpoint.6390  checkpoint.6440  checkpoint.6490  checkpoint.6540  checkpoint.6590  checkpoint.6640  page.6146  page.6196  page.6246  page.6296  page.6346  page.6396  page.6446  page.6496  page.6546  page.6596  page.6646
checkpoint.6141  checkpoint.6191  checkpoint.6241  checkpoint.6291  checkpoint.6341  checkpoint.6391  checkpoint.6441  checkpoint.6491  checkpoint.6541  checkpoint.6591  checkpoint.6641  page.6147  page.6197  page.6247  page.6297  page.6347  page.6397  page.6447  page.6497  page.6547  page.6597  page.6647
checkpoint.6142  checkpoint.6192  checkpoint.6242  checkpoint.6292  checkpoint.6342  checkpoint.6392  checkpoint.6442  checkpoint.6492  checkpoint.6542  checkpoint.6592  checkpoint.6642  page.6148  page.6198  page.6248  page.6298  page.6348  page.6398  page.6448  page.6498  page.6548  page.6598  page.6648
checkpoint.6143  checkpoint.6193  checkpoint.6243  checkpoint.6293  checkpoint.6343  checkpoint.6393  checkpoint.6443  checkpoint.6493  checkpoint.6543  checkpoint.6593  checkpoint.6643  page.6149  page.6199  page.6249  page.6299  page.6349  page.6399  page.6449  page.6499  page.6549  page.6599  page.6649
checkpoint.6144  checkpoint.6194  checkpoint.6244  checkpoint.6294  checkpoint.6344  checkpoint.6394  checkpoint.6444  checkpoint.6494  checkpoint.6544  checkpoint.6594  checkpoint.6644  page.6150  page.6200  page.6250  page.6300  page.6350  page.6400  page.6450  page.6500  page.6550  page.6600  page.6650
checkpoint.6145  checkpoint.6195  checkpoint.6245  checkpoint.6295  checkpoint.6345  checkpoint.6395  checkpoint.6445  checkpoint.6495  checkpoint.6545  checkpoint.6595  checkpoint.6645  page.6151  page.6201  page.6251  page.6301  page.6351  page.6401  page.6451  page.6501  page.6551  page.6601  page.6651
checkpoint.6146  checkpoint.6196  checkpoint.6246  checkpoint.6296  checkpoint.6346  checkpoint.6396  checkpoint.6446  checkpoint.6496  checkpoint.6546  checkpoint.6596  checkpoint.6646  page.6152  page.6202  page.6252  page.6302  page.6352  page.6402  page.6452  page.6502  page.6552  page.6602  page.6652
checkpoint.6147  checkpoint.6197  checkpoint.6247  checkpoint.6297  checkpoint.6347  checkpoint.6397  checkpoint.6447  checkpoint.6497  checkpoint.6547  checkpoint.6597  checkpoint.6647  page.6153  page.6203  page.6253  page.6303  page.6353  page.6403  page.6453  page.6503  page.6553  page.6603
checkpoint.6148  checkpoint.6198  checkpoint.6248  checkpoint.6298  checkpoint.6348  checkpoint.6398  checkpoint.6448  checkpoint.6498  checkpoint.6548  checkpoint.6598  checkpoint.6648  page.6154  page.6204  page.6254  page.6304  page.6354  page.6404  page.6454  page.6504  page.6554  page.6604
checkpoint.6149  checkpoint.6199  checkpoint.6249  checkpoint.6299  checkpoint.6349  checkpoint.6399  checkpoint.6449  checkpoint.6499  checkpoint.6549  checkpoint.6599  checkpoint.6649  page.6155  page.6205  page.6255  page.6305  page.6355  page.6405  page.6455  page.6505  page.6555  page.6605
checkpoint.6150  checkpoint.6200  checkpoint.6250  checkpoint.6300  checkpoint.6350  checkpoint.6400  checkpoint.6450  checkpoint.6500  checkpoint.6550  checkpoint.6600  checkpoint.6650  page.6156  page.6206  page.6256  page.6306  page.6356  page.6406  page.6456  page.6506  page.6556  page.6606
checkpoint.6151  checkpoint.6201  checkpoint.6251  checkpoint.6301  checkpoint.6351  checkpoint.6401  checkpoint.6451  checkpoint.6501  checkpoint.6551  checkpoint.6601  checkpoint.6651  page.6157  page.6207  page.6257  page.6307  page.6357  page.6407  page.6457  page.6507  page.6557  page.6607
checkpoint.6152  checkpoint.6202  checkpoint.6252  checkpoint.6302  checkpoint.6352  checkpoint.6402  checkpoint.6452  checkpoint.6502  checkpoint.6552  checkpoint.6602  checkpoint.head  page.6158  page.6208  page.6258  page.6308  page.6358  page.6408  page.6458  page.6508  page.6558  page.6608
checkpoint.6153  checkpoint.6203  checkpoint.6253  checkpoint.6303  checkpoint.6353  checkpoint.6403  checkpoint.6453  checkpoint.6503  checkpoint.6553  checkpoint.6603  page.6109        page.6159  page.6209  page.6259  page.6309  page.6359  page.6409  page.6459  page.6509  page.6559  page.6609
checkpoint.6154  checkpoint.6204  checkpoint.6254  checkpoint.6304  checkpoint.6354  checkpoint.6404  checkpoint.6454  checkpoint.6504  checkpoint.6554  checkpoint.6604  page.6110        page.6160  page.6210  page.6260  page.6310  page.6360  page.6410  page.6460  page.6510  page.6560  page.6610
checkpoint.6155  checkpoint.6205  checkpoint.6255  checkpoint.6305  checkpoint.6355  checkpoint.6405  checkpoint.6455  checkpoint.6505  checkpoint.6555  checkpoint.6605  page.6111        page.6161  page.6211  page.6261  page.6311  page.6361  page.6411  page.6461  page.6511  page.6561  page.6611
checkpoint.6156  checkpoint.6206  checkpoint.6256  checkpoint.6306  checkpoint.6356  checkpoint.6406  checkpoint.6456  checkpoint.6506  checkpoint.6556  checkpoint.6606  page.6112        page.6162  page.6212  page.6262  page.6312  page.6362  page.6412  page.6462  page.6512  page.6562  page.6612
checkpoint.6157  checkpoint.6207  checkpoint.6257  checkpoint.6307  checkpoint.6357  checkpoint.6407  checkpoint.6457  checkpoint.6507  checkpoint.6557  checkpoint.6607  page.6113        page.6163  page.6213  page.6263  page.6313  page.6363  page.6413  page.6463  page.6513  page.6563  page.6613
checkpoint.6158  checkpoint.6208  checkpoint.6258  checkpoint.6308  checkpoint.6358  checkpoint.6408  checkpoint.6458  checkpoint.6508  checkpoint.6558  checkpoint.6608  page.6114        page.6164  page.6214  page.6264  page.6314  page.6364  page.6414  page.6464  page.6514  page.6564  page.6614
bash-4.2$ du -sh data/queue/riemann
34G     data/queue/riemann
bash-4.2$

Because of this, logstash seems to be stuck trying to map all the queue files to memory without reporting errors or progress. It seems to be trying to mmap all the pages at once :blush:

You can find the entire debugging session and conversation here:

https://elastic.slack.com/archives/C0D8VG3ND/p1586860999060100

created time in a month

pull request commentansible-collections/community.general

Deprecated Helm module

Makes sense! I don't really have too much time available to maintain this package and I am not really using it anymore.

Thanks for taking care of the clean-up :hugs:

LucasBoisserie

comment created time in 2 months

issue commentcesanta/docker_auth

Scope Requests not returning correct values

One more update... I noticed if the requested resource includes the domain it rolls down to the catch all. If the requested resource is only namespace/name it returns the correct permissions.

curl -k --user fxdgear:hunter2 https://auth-server.com/auth\?account=fxdgear\&scope=repository:namespace/name:push,pull\&service=notary-server                                                                                                            

vs

curl -k --user fxdgear:hunter2 https://auth-server.com/auth\?account=fxdgear\&scope=repository:my-registry.com/namespace/name:push,pull\&service=notary-server

I think this may be a configuration issue on our end. The name extracted from the scope in the second example is my-registry.com/namespace/name, which according to the docker documentation is a valid name. Names in a scope can have (or ommit) the registry hostname.

The regular expression in the config file is /^namespace/${account}/, which fails in the case where the hostname is present because of the ^.

@fxdgear Let me know if the above makes sense to you or if I am missing something. I tried both curl commands with a modified config that didn't have the ^ and it seems to work as expected.

fxdgear

comment created time in 2 months

pull request commentopenshift/release

Drop flaper87 and tomassedovic from OWNERS_ALIASES

+1

tomassedovic

comment created time in 2 months

pull request commentkubernetes/cloud-provider-openstack

WIP:Add pagination support

/unassign @flaper87

ramineni

comment created time in 2 months

PR opened elastic/helm-charts

Reviewers
Accept a string as extraInitContainers value for filebeat

This makes the filebeat chart consistent with the other beats, while preserving backwards compatibility.

Fixes #490

  • [x ] Chart version not bumped (the versions are all bumped and released at the same time)
  • [x] README.md updated with any new values or changes
  • [x] Updated template tests in ${CHART}/tests/*.py
  • [x] Updated integration tests in ${CHART}/examples/*/test/goss.yaml
+28 -2

0 comment

3 changed files

pr created time in 2 months

push eventelastic/helm-charts

Flavio Percoco

commit sha dcd5828fbbd1c073b6211cbd10f14c590b5510be

Accept a string as extraInitContainers value for filebeat This makes the filebeat chart consistent with the other beats, while preserving backwards compatibility. Fixes #490

view details

push time in 2 months

create barnchelastic/helm-charts

branch : filebeat-extra-init

created branch time in 2 months

issue commentelastic/beats

Find kubernetes resource's root controller when including kubernetes metadata

@blakerouse thanks for the feedback. I'm working on a PR now but I'm unsure whether I'm implementing this the best way possible. :smile: We'll see when it is ready for review. :+1:

flaper87

comment created time in 2 months

issue commentelastic/beats

Find kubernetes resource's root controller when including kubernetes metadata

FWIW, I am happy to work on this myself (I actually already started playing around with it a bit) if someone would be willing to provide support while I do it. I'm not super familiar with the codebase or libbeats common practices :smile:

flaper87

comment created time in 2 months

issue openedelastic/beats

Find kubernetes resource's root controller when including kubernetes metadata

The current implementation of the add_kubernetes_metadata processor includes the Pod metadata and the immediate controller that is returned in the OwnerReferences list. This information is quite useful but it is not enough.

As part of https://github.com/elastic/infra/pull/17976 we added the option to compute "unique" service names for the events coming from a filebeat process running within a POD. Ideally, the service name would be unique per deployment but we don't have that information for events coming from a POD because the controller of the pod is the replicaset.

The service name we are computing right now is unique during the lifespan of the replicaset contorlling the PODs. Ideally, this shouldn't change after a rollout or upgrade, which is what using the deployment name would give us.

Describe the enhancement:

It would be great to have libbeats resolve the entire chain of controllers and return their metadata, which gives a better way to group events coming from pods based on the entire chain of resource controllers.

Describe a specific use case for the enhancement or feature:

  • Create unique service names from llama: https://github.com/elastic/infra/pull/17976
  • Create metrics, and visualizations for events coming from child resources using their parent resources.

created time in 2 months

delete branch elastic/beats

delete branch : fp/kubernetes-logging

delete time in 3 months

push eventelastic/beats

Flavio Percoco

commit sha 1d6323fb17427ecfe4d2e8d7967195fdb5c01be0

Improve some logging messages for add_kubernetes_metadata processor (#16866) Switch from Debug to Error when unrecoveral events happen and add extra debug messages when indexing and matching pods.

view details

push time in 3 months

PR merged elastic/beats

Improve some logging messages for add_kubernetes_metadata processor Team:Platforms [zube]: In Review
  • Enhancement

What does this PR do?

Switch from Debug to Error when unrecoveral events happen and add extra debug messages when indexing and matching pods.

I was trying to debug why my configs wouldn't work and I was forced to enable debug to notice that the add_kubernetes_metadata processor was failing because my configs were wrong.

Sadly, my configs weren't working even after I fixed the above and it was a bit painful to figure out what was going on because there weren't enough debug messages to understand what index keys, matches, and metadata was being processed by filebeat.

Why is it important?

It makes operations and debugging easier

Checklist

<!-- Mandatory Add a checklist of things that are required to be reviewed in order to have the PR approved

List here all the items you have verified BEFORE sending this PR. Please DO NOT remove any item, striking through those that do not apply. (Just in case, strikethrough uses two tildes. Scratch this.) -->

  • [x] My code follows the style guidelines of this project
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • [x] I have added tests that prove my fix is effective or that my feature works
+12 -5

4 comments

5 changed files

flaper87

pr closed time in 3 months

pull request commentelastic/beats

Improve some logging messages for add_kubernetes_metadata processor

I can't check the failures in great detail right now. Are these related to the PR or can I send the PR through?

flaper87

comment created time in 3 months

Pull request review commentelastic/beats

Improve some logging messages for add_kubernetes_metadata processor

 func (f *LogPathMatcher) MetadataIndex(event common.MapStr) string { 		logp.Debug("kubernetes", "Incoming log.file.path value: %s", source)  		if !strings.Contains(source, f.LogsPath) {-			logp.Debug("kubernetes", "Error extracting container id - source value does not contain matcher's logs_path '%s'.", f.LogsPath)+			logp.Err("Error extracting container id - source value does not contain matcher's logs_path '%s'.", f.LogsPath)

thank you! 😊

flaper87

comment created time in 3 months

push eventelastic/beats

Noémi Ványi

commit sha e5322af4f1e81d5b7692b4548c91d181e189590e

Install mage and goimports from vendor folder (#16836)

view details

Ivan Fernandez Calvo

commit sha 85f5d70f696900bb8befc43ba72a9c46e9a630ef

ci: avoid to modify go.mod file (#16842) * fix: avoid to modify go.mod file * test: do we need govendor? * fix: disable modules also when install mage * Update .ci/scripts/install-tools.bat * remove govendor code * Update .ci/scripts/install-tools.bat

view details

Andrew Wilkins

commit sha bbf9d6697f4a308b44484aca94359b347ebd1413

mage: optionally mount module cache for crossbuild (#16837) * mage: optionally mount module cache for crossbuild Introduce another mage variable, CrossBuildMountModcache, which defaults to false. When set to true, the host's module cache ($GOPATH/pkg/mod) will be mounted into the crossbuild Docker containers, read-only. To ensure the cache is up-to-date, we run "go mod download" on the host before starting the Docker containers. Also, fix buildGoDaemon to stop assuming that tsg/go-daemon is vendored. Instead, use "go list -m github.com/tsg/go-daemon" to find the directory in either the vendor directory or the module cache. Rather than having arguments for ListModulePath, create separate functions for listing in the module cache and vendor directory. Create an unexported function in dev-tools/mage which calls the appropriate one depending on the value of UseVendor. We now use this in "findElasticBeatsDir". Inline defaultConfigFileParams, otherwise the OSSBeatDir and LibbeatDir calls end up calling ElasticBeatsDir at package init time, before apm-server uas an opportunity to set UseVendor=false.

view details

Noémi Ványi

commit sha 420b717c74660774231bbfe6a46477f3035586ac

Fix link in an entry of CHANGELOG.next.asciidoc (#16872)

view details

Blake Rouse

commit sha 5c3891277f3a1fc76342125df79b5e081c47f72e

Fix NewContainerMetadataEnricher to use default config for kubernetes module. (#16857) * Fix code to use default config for resource metadata. * Add changelog entry.

view details

Flavio Percoco

commit sha b99ca7d057a3253999fa410e05969633722fbbce

Merge branch 'master' into fp/kubernetes-logging

view details

push time in 3 months

pull request commentelastic/beats

Improve some logging messages for add_kubernetes_metadata processor

@flaper87 Change looks good. Need to add a changelog entry.

Done, thanks for the review :)

flaper87

comment created time in 3 months

push eventelastic/beats

Flavio Percoco

commit sha 37156816655a55157d3f395d8811d15e8456af6c

Improve some logging messages for add_kubernetes_metadata processor Switch from Debug to Error when unrecoveral events happen and add extra debug messages when indexing and matching pods.

view details

push time in 3 months

push eventelastic/beats

Flavio Percoco

commit sha a41cb961abcc76c32b3661e628d7ca424e2bd9e2

Improve some logging messages for add_kubernetes_metadata processor Switch from Debug to Error when unrecoveral events happen and add extra debug messages when indexing and matching pods.

view details

push time in 3 months

push eventelastic/beats

Pier-Hugues Pellerin

commit sha ab84afc9782d43685d1fbaae698dc334b79f4307

Agent required changes for building the agent artifacts (#16624) * Agent required changes for building the agent artifacts The Agent has special requirements: - Only Elastic licensed - Bundle Metricbeat - Bundle Filebeat This PR isolate all the required changes for the devtools to correctly build the agent. This can be merged as is because it doesn't impact building any other beats. * Update dev-tools/mage/pkgspecs.go Co-Authored-By: Nicolas Ruflin <spam@ruflin.com> * Update dev-tools/mage/pkgspecs.go Co-Authored-By: Nicolas Ruflin <spam@ruflin.com> * Update dev-tools/mage/pkgspecs.go Co-Authored-By: Nicolas Ruflin <spam@ruflin.com> * update min version for macos to 1.11 Co-authored-by: Nicolas Ruflin <spam@ruflin.com>

view details

Pier-Hugues Pellerin

commit sha a25903be9c11287957aff9dd12d58794e8bc95c2

Vendor Agent dependencies (#16616) * Vendor Agent dependencies This PR is only the dependencies required by the Agent. * Fix missing/bad ecslog dependencies

view details

Lee Hinman

commit sha 342f0e0379c101b5698a7c12781abd84a6b2d884

Improve ECS categorization field mappings in iis module (#16618) - event.category - event.kind - event.outcome - event.type - related.ip - related.user - lowercase http.request.method - improve grok in access pipeline - change access and error pipelines to yaml Closes #16165

view details

Blake Rouse

commit sha 07b03a7f58e0a8a00942003abb9316bf084cc34e

Add cloudfoundry module to metricbeat (#16671) * Split up parts into different metricsets. * Add value and container metricsets. * Add documentation. * Revert metricbeat.yml. * Run make update. * Fix documentation add changelog. * Fix bad change in azure module. * Run make update. * Improve reference config. * Document Kibana Space ID setting (#16622) * Document Kibana Space ID setting * Update reference YML files * Update libbeat/docs/shared-kibana-config.asciidoc Co-Authored-By: DeDe Morton <dede.morton@elastic.co> * Updating reference YML files * Adding link to doc Co-authored-by: DeDe Morton <dede.morton@elastic.co> * Remove the special pins type for `ca_sha256` (#16615) We have changed the type from a special custom type to a slice of string. This reduce the number of exposed types and make it easier in general to work with in the tests. * Remove feature.Describer and enhance Details (#16658) * Make memqueue.Broker internal (now memqueue.broker) (#16667) * [docs] Add skeleton files for 7.7 release highlights and breaking changes (#16584) * [Metricbeat] Add vpc metricset for aws module (#16111) * Add vpn into vpc metricset * Add NATGateway and TrasitGateway * Update cisco.asciidoc (#16434) * Update cisco.asciidoc Making Elasticsearch explicit in the instructions * Update docs source in the module dir Co-authored-by: Loek van Gool <loek.vangool@elastic.co> * Fix wording in dev guide (#16497) * Added NetBeat (#12612) * Close files earlier when generating fields files (#16681) Fields generator keeps all open files while generating the global fields.yml file. This is not needed and may lead to "too many open files" errors in some environments. * [Metricbeat] Add new module for Redis Enterprise (#16482) Add metricsets: node, proxy * Add doppler_address, uaa_address, rlp_address to config.reference.yml. Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com> Co-authored-by: Steffen Siering <steffen.siering@elastic.co> Co-authored-by: Fae Charlton <fae.charlton@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Loek van Gool <loek.vangool@elastic.co> Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com> Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>

view details

Lee Hinman

commit sha e7f933588e9d1f9a58cf7e9acfaf03f2944da49f

Improve ECS categorization field mapping in kafka module (#16645) - event.kind - event.type - convert pipeline to yaml Closes #16167

view details

Noémi Ványi

commit sha 95626b8f1690344312c0831ab2bdcbccffe4d089

Go modules (#15853) This PR introduces the new dependency management system for beats. From now on we are using `go mod` instead of `govendor`. The name of the module we provide is named `github.com/elastic/beats/v7`. We adopted go modules, but we still keep dependencies under the folder `vendor`. However, it is maintained by `go mod` now by running `mage vendor`. Thus, it is not possible to add local changes to the dependencies there because next time someone runs the command it will be overwritten. **If you need to apply patches to dependencies either fork the repository or try to contribute it back to the original repo.** This PR does not address the changes in Beat generators. Those are going to be updated in a follow-up PR. ### FAQ #### How can I add a new dependency? TL;DR ```sh go get {{ module_name }}@{{ required_version }} mage vendor make notice ``` Run `go get {{ module_name }}@{{ required_version }}` in the repository. This adds the module to the requirements list in `go.mod`. In order to add the dependency to the vendor folder, run `mage vendor`. If the dependency contains files which are not copied by this command (e.g. C source files), add them to this list: https://github.com/elastic/beats/tree/master/dev-tools/mage/gomod.go #### How can I upgrade a dependency? The process is similar to adding a new dependency. Except for one step. If the dependency is updated to a newer major version, make sure to follow up changes in the root import paths of those libs. I suggest using the tool named `mod` to upgrade it: https://github.com/marwan-at-work/mod #### How can I use a fork of a dependency? If you need to use a fork of a dependency, add it to `go.mod` either manually or by running the following command: ```sh go mod edit -replace {{ dependency_name }}={{ fork_name }}@{{ fork_version }} ``` #### Why is the major version part of the import path? In a nutshell, it is needed to tell apart major version changes. Their import paths are different because those are incompatible. See more: https://blog.golang.org/v2-go-modules Thus, when we are releasing v8.0.0, we need to change the import path again. The tool named mod is going to be useful for us in this case, too. #### Why is `make check` failing? Are you seeing the following error? ``` make[1]: Leaving directory `/home/travis/gopath/src/github.com/elastic/beats' diff --git a/go.mod b/go.mod index 170f6660..fc5fb6ee 100644 --- a/go.mod +++ b/go.mod @@ -53,6 +53,7 @@ require ( github.com/dop251/goja_nodejs v0.0.0-20171011081505-adff31b136e6 github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4 github.com/eclipse/paho.mqtt.golang v1.2.1-0.20200121105743-0d940dd29fd2 + github.com/elastic/beats v7.6.0+incompatible github.com/elastic/ecs v1.4.0 github.com/elastic/go-libaudit v0.4.0 github.com/elastic/go-licenser v0.2.1 ``` If your `go.mod` lists `github.com/elastic/beats` as a requirement, the repo contains an outdated root import path. Change `github.com/elastic/beats` to `github.com/elastic/beats/v7`. #### What is next? After this PR is merged, please rebase all of your open PRs. In most cases rebasing will lead to conflicts. Possible sources of conflicts: 1. New import path conflicts with the old. To resolve it switch every `github.com/elastic/beats` import to `github.com/elasitc/beats/v7`. 2. A new dependency has been added or existing was updated. Update the dependency using go modules. ### Further reading for the interested - [Golang blog: Using Go modules](https://blog.golang.org/using-go-modules) - [Golang wiki](https://github.com/golang/go/wiki/Modules) ### Related issues elastic/beats#15868

view details

Marcin Tojek

commit sha cdadced5d5172441ccda528ce6b8f241ef22be9e

Wait for emitted metrics (#16721)

view details

Mariana Dima

commit sha d89675fc3bc4f6a8e72671ed887bf96ad8ca159b

Add container azure metricset (#16421) * add container * update changelog * fix test * separate container * separate * regenerate * fixes * tests * fix test

view details

kaiyan-sheng

commit sha 10ed037b8e42331a1842cb4136668ef57920776a

Add documentation for tags_filter and tags (#16731)

view details

Mariana Dima

commit sha 8b547972a26ff8fb94bbcd0904e96363ace9f2df

Fix imports for previous PR (#16756) * fix * changelog

view details

Lee Hinman

commit sha 40762113e80e3e74c3b36b9a31465ee83698e1f7

[Filebeat] Improve ECS categorization field mappings in kibana module (#16652) * Improve ECS categorization field mappings in kibana module - event.kind - event.outcome - event.type - convert pipeline to yaml Closes #16168

view details

Andrew Stucki

commit sha 2c310c09e43f6aa23311ecb48142a8a247fcd32b

Fix zeek pipelines to include millisecond precision (#16766) * Fix zeek pipelines to include millisecond precision * Add changelog entry

view details

Marcin Tojek

commit sha 07a9c754e6035cef75331c6d4c3b5e270b9e2cb2

[Metricbeat] Add dashboard for redisenterprise (#16752) * Add dashboard for redisenterprise * Update screenshot

view details

Jameel Al-Aziz

commit sha 6785901e609ee6e898b5a51b681b2746d480a61c

Fix renaming cloud metadata processor fields (#13808) Clone the underlying metadata to allow renaming sub-fields under cloud that are injected by the cloud metadata processor.

view details

DeDe Morton

commit sha d9f553dd440d2fefca5db279d41511674cb0279e

[docs] Add username and password to haproxy example (#16703)

view details

Steffen Siering

commit sha 9db5dc3eb9cf3e60136aa9a25008095606a02174

Move crawler to filebeat/beater (#16700) The Crawler is more or less the main run loop of filebeat. It used to generate the factories for inputs and modules, yet did expose those via public interfaces, so we can hook them up with autodiscovery. The crawler constuctor used to create only a half-instantiated object, requiring all additional setup to be run after crawler.Start, as start will finally initialized the shared variables. This change moves the crawler into filebeat and makes it private (it is not used anywhere in the Beats codebase). In addition to this, we now create the factories upfront and pass them as proper dependencies to autodiscovery and the crawler.

view details

Jaime Soriano Pastor

commit sha 10da237d9b60a246662c512d91f241d2551d956e

Fix and extend logging of light modules load errors (#14706) * Fix and extend logging of light modules load errors * Add BEAT_STRICT_PERMS to beat container in metricbeat docker compose * Add changelog * Use local logger, and fix error messages * Ignore errors if module directory doesn't exist * Fix issue when modules directory contains regular files * Remove empty line * Add test for issue with files listed as modules

view details

Adrian Serrano

commit sha 68e9bd33fa19659fc7e1f2a298acf27c8007788d

[Auditbeat] macOS: Replace file.origin.raw with file.origin.text (#15630) The `raw` part of the multifield was unnecessary because it was keyword like the base field. Replaced with `file.origin.text` of type text as ECS recommends. Fixes #12423

view details

kaiyan-sheng

commit sha a47a49124a0f345de1a47bb174b2b798bf7f17e8

Fix documentation typo for kafka metricbeat module (#16772)

view details

Shaunak Kashyap

commit sha f6869827a18b8ae0feb8aa5956f4042502675241

Generate haproxy module docs (#16776)

view details

push time in 3 months

PR opened elastic/beats

Improve some logging messages for add_kubernetes_metadata processor
  • Enhancement

What does this PR do?

Switch from Debug to Error when unrecoveral events happen and add extra debug messages when indexing and matching pods.

I was trying to debug why my configs wouldn't work and I was forced to enable debug to notice that the add_kubernetes_metadata processor was failing because my configs were wrong.

Sadly, my configs weren't working even after I fixed the above and it was a bit painful to figure out what was going on because there weren't enough debug messages to understand what index keys, matches, and metadata was being processed by filebeat.

Why is it important?

It makes operations and debugging easier

Checklist

<!-- Mandatory Add a checklist of things that are required to be reviewed in order to have the PR approved

List here all the items you have verified BEFORE sending this PR. Please DO NOT remove any item, striking through those that do not apply. (Just in case, strikethrough uses two tildes. Scratch this.) -->

  • [ ] My code follows the style guidelines of this project
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • [ ] I have added tests that prove my fix is effective or that my feature works
+112 -106

0 comment

78 changed files

pr created time in 3 months

create barnchelastic/beats

branch : fp/kubernetes-logging

created branch time in 3 months

issue commentelastic/cloud-on-k8s

Relabel file extension should be yml

ok, it's fixed!

flaper87

comment created time in 3 months

push eventelastic/cloud-on-k8s

Flavio Percoco

commit sha 7485ed0a6aa145207431a43b798ca3c7496640ba

Let's use a stricter rule for relabel configs

view details

push time in 3 months

issue commentelastic/cloud-on-k8s

Relabel file extension should be yml

mmh, there seems to be something wrong here.

flaper87

comment created time in 3 months

IssuesEvent

PR opened elastic/cloud-on-k8s

rename relabel file to .yml

Fixes #2666

+0 -0

0 comment

1 changed file

pr created time in 3 months

push eventelastic/cloud-on-k8s

Flavio Percoco

commit sha bae51483702057d2bfcce5c1ddeef03f9141ec77

rename relabel file to .yml Fixes #2666

view details

push time in 3 months

create barnchelastic/cloud-on-k8s

branch : yml-not-yaml

created branch time in 3 months

issue openedelastic/cloud-on-k8s

This is a relabel test!

created time in 3 months

more