profile
viewpoint
Andrew Pogrebnoy dAdAbird @percona Limassol, Cyprus

percona/percona-backup-mongodb 142

Percona Backup for MongoDB

Percona-Lab/percona-openshift 52

Set of scripts to run Percona software in OpenShift / Kubernetes

akira-kurogane/percona-backup-mongodb 0

Percona Backup for MongoDB

dAdAbird/ClickHouse 0

ClickHouse is a free analytic DBMS for big data.

dAdAbird/cockroach 0

CockroachDB - the open source, cloud-native SQL database.

dAdAbird/docker 0

Docker - the open-source application container engine

dAdAbird/easyjson 0

Fast JSON serializer for golang.

dAdAbird/etcd 0

A distributed consistent key-value store for shared configuration and service discovery

dAdAbird/fasthttp 0

Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http

push eventpercona/percona-backup-mongodb

Damiano Albani

commit sha 40897fd6edc884e0b64a6d2d44891c27be4f467a

[PBM-764] Support Zstandard compression format (#622) * Add support for Zstandard compression * Set compression level for Zstandard * Add "zstd" to the list of compression types accepted by commands https://jira.percona.com/browse/PBM-764

view details

push time in a day

PR merged percona/percona-backup-mongodb

[PBM-764] Support Zstandard compression format

https://jira.percona.com/browse/PBM-764

+30 -12

4 comments

10 changed files

dalbani

pr closed time in a day

PullRequestReviewEvent

push eventpercona/percona-backup-mongodb

Damiano Albani

commit sha 6e23cc9bc902f9f595409dd37d280c6dd86bcce5

Support compression settings for PITR chunks (#626) https://jira.percona.com/browse/PBM-770

view details

push time in a day

PullRequestReviewEvent
PullRequestReviewEvent

push eventpercona/percona-backup-mongodb

Andrew Pogrebnoy

commit sha e2b6047ea45692f0ab61d08f9dc775f6b4c7aefb

Fix vesion typo

view details

push time in 14 days

pull request commentpercona/percona-backup-mongodb

[PBM-756] Support configurable compression level for backups

By the way, why isn't Jenkins triggered and the E2E tests not being run in case of a PR just like mine? @dalbani Due to security reasons, we don't trigger Jenkins with the code from external contributors. We mulling thru options on tests triggered manually by maintainers etc. But this is a low priority task for the time being (despite it being a wee stone in the shoe)

dalbani

comment created time in 18 days

delete branch percona/percona-backup-mongodb

delete branch : defbin/fix-aquire-typo

delete time in 18 days

push eventpercona/percona-backup-mongodb

Dmytro Zghoba

commit sha 9f24f63a197a931f247891436b5b98a0bd667a6f

typo: "aquire" is a misspelling of "acquire". use "acquire" for non-exported resources (#616)

view details

push time in 18 days

PullRequestReviewEvent

Pull request review commentpercona/percona-backup-mongodb

[PBM-764] Support Zstandard compression format

 func Compress(w io.Writer, compression pbm.CompressionType) io.WriteCloser { 			cc = 1 		} 		return s2.NewWriter(w, s2.WriterConcurrency(cc))+	case pbm.CompressionTypeZstandard:+		zw, _ := zstd.NewWriter(w)

Hi @dalbani ! Compress() doesn't return any errors since nothing inside it may trigger any. zw, _ := zstd.NewWriter(w) won't return errors since there unless it takes some options. I just wanted to express that in the comment to explain why the error isn't handled there. But, given that we have https://github.com/percona/percona-backup-mongodb/pull/619 which brings Compress() (io.WriteCloser, error) I'd rather wait for the 619 to be merged. And then add zstd along with the compression level. If this makes sense to you.

And can you also add it to the pbm-speed-test.

Thanks!

dalbani

comment created time in 18 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentpercona/percona-backup-mongodb

[PBM-756] Support configurable compression level for backups

 func Run(nodeCN *mongo.Client, stg storage.Storage, compression pbm.CompressionT  	r := &Results{} 	ts := time.Now()-	size, err := backup.Upload(context.Background(), src, stg, compression, fileName, -1)+	size, err := backup.Upload(context.Background(), src, stg, compression, nil, fileName, -1)

It makes sense to add compression level to the speed-test as well

dalbani

comment created time in 18 days

Pull request review commentpercona/percona-backup-mongodb

[PBM-756] Support configurable compression level for backups

 func runBackup(cn *pbm.PBM, b *backupOpts, outf outFormat) (fmt.Stringer, error) 		return nil, errors.Wrap(err, "get remote-store") 	} +	var level *int

I propose using -1 as the default. kinping's flags have Default() functions so we're safe here. The only potential pitfall is the straightforward usage of BackupCMD But we can add some comments there so a user would be aware of default bechaviour.

BTW, I took a look and pgzip's DefaultCompression is -1 as well

dalbani

comment created time in 18 days

Pull request review commentpercona/percona-backup-mongodb

[PBM-756] Support configurable compression level for backups

 type NopCloser struct { func (NopCloser) Close() error { return nil }  // Compress makes a compressed writer from the given one-func Compress(w io.Writer, compression pbm.CompressionType) io.WriteCloser {+func Compress(w io.Writer, compression pbm.CompressionType, level *int) (io.WriteCloser, error) { 	switch compression { 	case pbm.CompressionTypeGZIP:-		return gzip.NewWriter(w)+		if level == nil {+			level = aws.Int(gzip.DefaultCompression)+		}+		gw, err := gzip.NewWriterLevel(w, *level)+		if err != nil {+			return nil, err+		}+		return gw, nil 	case pbm.CompressionTypePGZIP:-		pgw := pgzip.NewWriter(w)+		if level == nil {+			level = aws.Int(pgzip.DefaultCompression)+		}+		pgw, err := pgzip.NewWriterLevel(w, *level)+		if err != nil {+			return nil, err+		} 		cc := runtime.NumCPU() / 2 		if cc == 0 { 			cc = 1 		}-		pgw.SetConcurrency(1<<20, cc)-		return pgw+		err = pgw.SetConcurrency(1<<20, cc)+		if err != nil {+			return nil, err+		}+		return pgw, nil 	case pbm.CompressionTypeLZ4:-		return lz4.NewWriter(w)+		lz4w := lz4.NewWriter(w)+		if level != nil {+			lz4w.Header.CompressionLevel = *level+		}+		return lz4w, nil 	case pbm.CompressionTypeSNAPPY:-		return snappy.NewBufferedWriter(w)+		return snappy.NewBufferedWriter(w), nil 	case pbm.CompressionTypeS2: 		cc := runtime.NumCPU() / 3 		if cc == 0 { 			cc = 1 		}-		return s2.NewWriter(w, s2.WriterConcurrency(cc))+		writerOptions := []s2.WriterOption{s2.WriterConcurrency(cc)}+		if level != nil {+			switch *level {+			case 1:+				writerOptions = append(writerOptions, s2.WriterUncompressed())+				break

no need in break. In Go control doesn't fall to the next case by default (https://go.dev/ref/spec#Switch_statements)

dalbani

comment created time in 18 days

PullRequestReviewEvent

Pull request review commentpercona/percona-backup-mongodb

[PBM-764] Support Zstandard compression format

 func Compress(w io.Writer, compression pbm.CompressionType) io.WriteCloser { 			cc = 1 		} 		return s2.NewWriter(w, s2.WriterConcurrency(cc))+	case pbm.CompressionTypeZstandard:+		zw, _ := zstd.NewWriter(w)
		// an error is always nil with no options
		zw, _ := zstd.NewWriter(w)
dalbani

comment created time in 18 days

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentpercona/percona-backup-mongodb

PBM-734: add storage.s3.debugLogLevels option

 func (p *PBM) SetConfig(cfg Config) error { 		if err != nil { 			return errors.Wrap(err, "cast storage") 		}++		s3.SDKLogLevel(cfg.Storage.S3.DebugLogLevels, os.Stderr)
		// in sake of options verification. warnings will be printed into stderr
		s3.SDKLogLevel(cfg.Storage.S3.DebugLogLevels, os.Stderr)
defbin

comment created time in 19 days

Pull request review commentpercona/percona-backup-mongodb

PBM-734: add storage.s3.debugLogLevels option

 func (c *Conf) Cast() error { 	return nil } +// SDKLogLevel returns AWS SDK log level value from comma-separated+// SDKDebugLogLevel values string. If the string is empty, returns aws.LogOff.+//+// If the string is incorrect formatted, prints warnings to the io.Writer.+// Passing nil as the io.Writer will discard any warnings.+func SDKLogLevel(levels string, out io.Writer) aws.LogLevelType {+	if levels == "" {+		return aws.LogOff+	}++	if out == nil {+		out = ioutil.Discard+	}++	var logLevel aws.LogLevelType+	ss := make(map[string]struct{})++	for _, lvl := range strings.Split(levels, ",") {+		l := SDKDebugLogLevel(lvl).SDKLogLevel()+		if l == 0 {+			fmt.Fprintf(out, "WARN: S3 client debug log level: unsupported %q\n", lvl)+			continue+		}++		if _, ok := ss[lvl]; ok {+			fmt.Fprintf(out, "WARN: S3 client debug log level: duplicated %q\n", lvl)

s/WARN/Warning In other places in CLI we are using the"Error:.." format. "Warning: ..." would be more consistent

defbin

comment created time in 19 days

Pull request review commentpercona/percona-backup-mongodb

PBM-734: add storage.s3.debugLogLevels option

 func (c *Conf) Cast() error { 	return nil } +// SDKLogLevel returns AWS SDK log level value from comma-separated+// SDKDebugLogLevel values string. If the string is empty, returns aws.LogOff.+//+// If the string is incorrect formatted, prints warnings to the io.Writer.+// Passing nil as the io.Writer will discard any warnings.+func SDKLogLevel(levels string, out io.Writer) aws.LogLevelType {+	if levels == "" {+		return aws.LogOff+	}++	if out == nil {+		out = ioutil.Discard+	}++	var logLevel aws.LogLevelType+	ss := make(map[string]struct{})++	for _, lvl := range strings.Split(levels, ",") {+		l := SDKDebugLogLevel(lvl).SDKLogLevel()+		if l == 0 {+			fmt.Fprintf(out, "WARN: S3 client debug log level: unsupported %q\n", lvl)+			continue+		}++		if _, ok := ss[lvl]; ok {+			fmt.Fprintf(out, "WARN: S3 client debug log level: duplicated %q\n", lvl)+			continue+		}++		ss[lvl] = struct{}{}

No need for checks and warnings for duplicates. Just skip it. Just do straight logLevel |= l if it's duplicate nothing will change.

defbin

comment created time in 19 days

Pull request review commentpercona/percona-backup-mongodb

PBM-734: add storage.s3.debugLogLevels option

 func (c *Conf) Cast() error { 	return nil } +// SDKLogLevel returns AWS SDK log level value from comma-separated+// SDKDebugLogLevel values string. If the string is empty, returns aws.LogOff.+//+// If the string is incorrect formatted, prints warnings to the io.Writer.+// Passing nil as the io.Writer will discard any warnings.+func SDKLogLevel(levels string, out io.Writer) aws.LogLevelType {+	if levels == "" {+		return aws.LogOff+	}++	if out == nil {+		out = ioutil.Discard+	}++	var logLevel aws.LogLevelType+	ss := make(map[string]struct{})++	for _, lvl := range strings.Split(levels, ",") {

I'd also trim spaces and skip empty options. So in the case of LogDebug, ,, HTTPBody , RequestRetries we'd have valid: LogDebug, HTTPBody, RequestRetries.

And no warning is needed in case of an empty option. Just skip it.

defbin

comment created time in 19 days

PullRequestReviewEvent
PullRequestReviewEvent

push eventpercona/percona-backup-mongodb

Andrew Pogrebnoy

commit sha 1fc28ebb2908c51ca06c8d04690eba2ac665c140

Fix cancel-backup & bump version

view details

push time in 19 days

more