profile
viewpoint

google/wuffs 1779

Wrangling Untrusted File Formats Safely

nigeltao/taowm 75

The Acutely Opinionated Window Manager

nigeltao/nstcc 15

The Not So Tiny C Compiler

nigeltao/fontscripts 1

Miscellaneous font-related scripts. Probably not very interesting to anyone but me.

nigeltao/snappy-test-data 1

Mirror of google/snappy's testdata directory

nigeltao/superswitcher 0

Automatically exported from code.google.com/p/superswitcher

nigeltao/zlib 0

A massively spiffy yet delicately unobtrusive compression library.

PR closed google/wuffs

Attemps to pass frame buffer data to browser context

Attempting to follow this article https://dev.to/chromiumdev/fast-gif-parsing-on-the-web-with-wasm--wuffs-48l4

+85 -67

2 comments

1 changed file

jfurrow

pr closed time in 2 hours

pull request commentgoogle/wuffs

Attemps to pass frame buffer data to browser context

Attempting to follow this article https://dev.to/chromiumdev/fast-gif-parsing-on-the-web-with-wasm--wuffs-48l4

Whilst that dev.to article ports Wuffs to Wasm, Wuffs does not primarily target Wasm, and this PR as is comments out a bunch of code that is actually used by the example program, as built by e.g. https://github.com/google/wuffs/blob/master/build-example.sh

See the screencast at https://github.com/google/wuffs/tree/master/example/gifplayer for the existing functionality.

jfurrow

comment created time in 2 hours

push eventgoogle/wuffs

Nigel Tao

commit sha a540fbce4ef5e9b91ecb6a20d6ce8aabd0c73259

Add test/data/file-sizes.json

view details

Nigel Tao

commit sha 7d31462e63dfb0dccd77630d8a6fa4eb5fa87336

Add std/json benches

view details

Nigel Tao

commit sha 51b1258b61754cfc617013ef30a8addc3ebcf4e0

Rename test/c throughput_counter enums

view details

Nigel Tao

commit sha 2914bae3b278d7b645fe14ddd1e3de46d2ad781c

Re-format C function signatures

view details

Nigel Tao

commit sha 31e5365a26830420574bd920f9f976f77452971a

Rename testlib's BUFFER_SIZE constant

view details

Nigel Tao

commit sha b319639badf669ede6d0d91d2e565c5270ec69e9

Add doc/note/auto-formatting.md

view details

Nigel Tao

commit sha 014b935d95ebaca105c0096798e89df1e3fc57bd

Add base.token_decoder interface

view details

Nigel Tao

commit sha f31819bed4d4f2e16533dca1bb4a16dcdda17857

Remove testlib's "#ifdef WUFFS_INCLUDE_GUARD"

view details

Nigel Tao

commit sha 2278a47cc6e10a94dc0ed889f3d879f1f587c111

Fix wuffs_base__parse_number_i64 goto-crosses-init

view details

Nigel Tao

commit sha 1da8110462ee61532f87caeb8346f733d2f0a6c8

Rename test/c test_cases loop variable

view details

Nigel Tao

commit sha 9d4eeb7b54203365d1e30fb92f603ffeab7fab6b

Tweak wuffs_base__token bitmask assignments again

view details

Nigel Tao

commit sha 9f0eb266e8e202789df65cfeb23addc1f00d9df1

Rename std/json Look-Up Table consts

view details

Nigel Tao

commit sha 3b17d5b28a6dd0fceb638c2258d3848e9845a514

Have std/json reject some invalid UTF-8

view details

Nigel Tao

commit sha 2e62fa369ab3f7d99b07bf8f2ed1b3e41ac1aa5d

Fix wuffs_base__parse_number_i64 warning

view details

Nigel Tao

commit sha 550ef90114fd63c64995a610d1fa5699918393db

Have std/json reject invalid UTF-8

view details

push time in 9 hours

push eventgoogle/wuffs

Nigel Tao

commit sha 2d2a880c189e8563f4d4cba7866cde5b1ba340f4

Ban unlabeled break/continue for labeled loop

view details

Nigel Tao

commit sha e6f34fe8a9dba31bb804c3c42a3de86574aaad2b

Rename cgen's break/continue labels

view details

Nigel Tao

commit sha ce50a39f8515cbd0f794213db37d7a0bf94aa863

Rename s/bad/fail/ in test/c

view details

Nigel Tao

commit sha 7b199116b08690e4ae6691e741f8c9661241fa02

Rename s/got/have/ in test/c

view details

Nigel Tao

commit sha 16b0c4652afe2d3cd5068b71cd65b1b3a9413f20

Have std/json decode Unicode surrogate pairs

view details

Nigel Tao

commit sha a9d1488674bdca0e3668e30f7cff38830d39a480

Tweak wuffs_base__token bitmask assignments

view details

Nigel Tao

commit sha 6d698a3a47c238390ca47c07a321d11316ae57ba

Add wuffs_base__parse_number_i64

view details

Nigel Tao

commit sha 29aadc6ff5fd436473b95bc8345c763fc3de43ed

Add more test/data/*.json files Original URLs: australian-abc-local-stations.json http://data.gov.au/geoserver/abc-local-stations/wfs?request=GetFeature&typeName=ckan_d534c0e9_a9bf_487b_ac8f_b7877a09d162&outputFormat=json nobel-prizes.json http://api.nobelprize.org/v1/prize.json

view details

push time in 2 days

issue commentgolang/go

x/image/font/gofont/gomedium: wrong shape for "l" letter

They were busy, but have coincidentally become less so recently. We'll hopefully release a fix soon.

MichaelMonashev

comment created time in 3 days

push eventgoogle/wuffs

Nigel Tao

commit sha 01bab829700bf88b8045c54c2479e4f624f8647d

Have std/json pushes be (from, to) not just (to)

view details

Nigel Tao

commit sha 9f7a250cf2ff2e668d8a8d238f58a54b0853fe91

Define WUFFS_BASE__TOKEN__VBC__ETC macros

view details

Nigel Tao

commit sha b1af7b9b411f088b5ca6660ddfba4e3fb4f2fc91

Add base/strconv-impl.c table column comments

view details

Nigel Tao

commit sha 9cc2c25f3dc85d69812e51a2cf4f1e2d520b2f8e

Have different exit codes for errors vs exceptions

view details

Nigel Tao

commit sha ea2cd1c5c93482b3760de777dc929561380a0f58

Fix lang/check test for endwhile syntax

view details

Nigel Tao

commit sha 569a294fc2a47c59df50d1fdca2918e0f0ecfc62

Add script/run-json-test-suite.sh

view details

Nigel Tao

commit sha 6b161af654bfa8dffb3be540687fb66d2ecb3240

Have example/jsonptr consume trailing whitespace

view details

Nigel Tao

commit sha 83ae50ce9bc71e74fe36f4da64cab561934b2e73

Raise std/json's max number length from 49 to 99

view details

Nigel Tao

commit sha c5b3a9e08ee1704a3db9e2ecb43b3d2e8ca1e5f1

Have std/json look at args.src.is_closed() more

view details

push time in 3 days

push eventgoogle/wuffs

Nigel Tao

commit sha c92c0f50e4b23f134801163d93454fe07c38a497

Have std/json decode null, false and true

view details

Nigel Tao

commit sha 93b09674a977850bf23dfe94805b9182e149b0e9

Add a "while.goto_foo true { etc }" to std/json

view details

Nigel Tao

commit sha b5461bdd5e8732dc1a7767c1f2985380a3867c87

Have std/json decode string escapes

view details

Nigel Tao

commit sha df62fe58461bab2d6e849172c63ab13edf6c1711

Fix hexadecimal_digits typo

view details

Nigel Tao

commit sha 4503f2de631e237f5b79fb22c6742eae5732c0e0

Add test/data/json-things.*.json

view details

Nigel Tao

commit sha 9263c80cfbc975a683414576f6675ccf2d381ee6

Factor out std/json/common_consts.wuffs

view details

Nigel Tao

commit sha 48e70046368e2fa24d4f0080af77df6b2d2b31c5

Add std/json character classes

view details

Nigel Tao

commit sha f6843719b5627a67505abd89f3998ec2121cef00

Re-order std/json if-else chain

view details

Nigel Tao

commit sha d1a4abe321472081cc893d476d684e3b31fe0d29

Refactor std/json expected-character-class check

view details

Nigel Tao

commit sha b1e3a15c0853ca573df47ab36f2e9fd97c3a0b50

Add endwhile syntax

view details

Nigel Tao

commit sha b2dd7d952c5f0237da7f9847cd8c7c6d7dc86859

Have io_reader.match7 check io_reader.is_closed

view details

push time in 4 days

push eventgoogle/wuffs

Nigel Tao

commit sha 70db96ac6d470fb161ffa2689bdb5ec905df4501

Add wuffs_base__parse_number_u64

view details

Nigel Tao

commit sha 9a5b6357788ee0ebec5b682d5b019dae2dc31e63

Have std/json distinguish int/float numbers

view details

push time in 6 days

push eventgoogle/wuffs

Nigel Tao

commit sha af666564bbc7f102d5f6a63955e6eab1403b7fab

Fix std/json stack bit mask

view details

Nigel Tao

commit sha 0711f23fed129dbeaefc63146cc5cec92e3c68f6

Fix missing yield inside std/json string decoder

view details

Nigel Tao

commit sha 27036043ae5de3a568c377a0341116633b0b60d7

Have std/json reject control characters in strings

view details

Nigel Tao

commit sha a8406927ef63267d800b92bf817b8347aa185dde

Add io_reader.is_closed

view details

Nigel Tao

commit sha 8850d38403837cda93bb738cabd92c5ef10f5168

Have std/json decode numbers

view details

Nigel Tao

commit sha 67306567f9676e5d0b0f430804b2f13e2a0e8fb7

Have std/json look at args.src.is_closed()

view details

Nigel Tao

commit sha 0aacdfd6e348491ccee1b311c3a8b8859daae66a

Allow r.skip32_fast!(actual: x, worst_case: x)

view details

Nigel Tao

commit sha 6a7a4d08e3cae5a6647ef4c8a6e5cd929625f1e8

Save derived vars for returns in non-coroutines

view details

Nigel Tao

commit sha 21c87ce6b3ae7141d73e91008be658b8f9b82aac

Skip writeSaveDerivedVar if func ends with return

view details

Nigel Tao

commit sha ae7062065b5b4c09868c1881c3057cb2b4969b54

Ban returning an impure expression

view details

Nigel Tao

commit sha 9d35cf00acdd2f9f6ce876fe26374a375f649eac

Tweak std/json's decode_tokens return values

view details

Nigel Tao

commit sha 7acb35f05d2fa3e8f730e00a14d02c49f60ef586

Have std/json decode fractions and exponents

view details

Nigel Tao

commit sha b194c7fbb96df3a342d3b4e6604bed477ffcdfeb

Have std/json check numbers are expected

view details

push time in 7 days

push eventgoogle/wuffs

Nigel Tao

commit sha 5c4e7fe126012912f3dba9232f3aebdd5cc656d7

Add base.token_reader to builtin.Types

view details

push time in 10 days

push eventgoogle/wuffs

Nigel Tao

commit sha 7a142168bec01db1fd6f0a9186fc34ed998ea3ae

Add std/json

view details

Nigel Tao

commit sha 737e31ff3a4f805e5765cc9b5db70b3cd3dfa666

Add tokens

view details

Nigel Tao

commit sha 2f2ccdb6964b2835f21288da73796378cb435224

Add token_writer.write_token

view details

Nigel Tao

commit sha 4c7c2960152b88d139f02059e23535671199013b

Strengthen liveness analysis around CSPs

view details

Nigel Tao

commit sha 8e7efa67b9737567ff2a5cbec936476a301b524b

Have write_{u8,token} save args across suspension

view details

Nigel Tao

commit sha 36857984c7ee4118d540bef568aaae73e8ba0655

Add wuffs_base__token methods

view details

Nigel Tao

commit sha e9aba8924a0d5d6b1c903fcdd8a340b7a42fd713

Add token_writer.write_fast_token

view details

Nigel Tao

commit sha ea91e5ab694eeab4b28233ad97777681915999da

Have std/json decode '[' and '{'

view details

Nigel Tao

commit sha 91454e911e680d6e408547341f41f0a50fef0cc4

Have std/json decode '"'

view details

Nigel Tao

commit sha 40778f0665d059fb5ed82bf22f176b5d3c9b0a23

Have std/json decode ',' and ':'

view details

Nigel Tao

commit sha 293e9a89283cfa0334e571beb974a10a1c90d1c4

Have std/json decode ']' and '}'

view details

Nigel Tao

commit sha 58a4aba4d4b686d6cbd77d6cdd919336033607c2

Have std/json pops show parent container type

view details

Nigel Tao

commit sha 1b073491441e5130707e371cbacc88b7bd92c7a0

Add example/jsonptr

view details

push time in 10 days

issue commentgolang/go

proposal: image: add generic metadata support for jpeg, gif, png

do you think there's anything we should do here?

@rsc, I don't think there's anything to do.

I agree with @shabbyrobe and @smasher164 to keep this issue open for the reasons they mentioned. The linked CLs (https://go-review.googlesource.com/c/go/+/208559 and https://go-review.googlesource.com/c/go/+/216799) are abandoned, but anyone from the future who wants to take a crack at fixing this issue can still read those CL discussions (and #27830 "proposal: image: decoding options").

drswork

comment created time in 14 days

push eventgoogle/wuffs

Nigel Tao

commit sha 096b7146f05c0b6492e228794a19151490c0dd28

Have wbmp decoder check for "#bad restart"

view details

Nigel Tao

commit sha 0853a716bb891767b58f9ec548317d408fa84738

Add std/bmp

view details

Nigel Tao

commit sha f69f849f11e5ed3fd647756feaeb6645dc1d0f9a

Implement bmp.decoder.decode_image_config

view details

Nigel Tao

commit sha 8378d9f3b49b8823a6d5506b132dd58392c4c5e0

Implement bmp.decoder.skip_frame

view details

Nigel Tao

commit sha e6cc2fe8f05f863ad4adf73f1b79ff6a8deac842

Implement bmp.decoder.decode_frame

view details

Nigel Tao

commit sha 8d29cb15ce0201cf31ccb88e488df2d397601c78

Re-order some image-impl.c multiplies

view details

Nigel Tao

commit sha fb9a0964a2d5174d1e510377ab9aba1f3f3c2739

Fix associative op const-value calculation

view details

Nigel Tao

commit sha 22be1c0da0f8aa905df6cfc67e167d51c2a2a075

Allow underscores in numeric constants

view details

Nigel Tao

commit sha b40671ec846a7603999c7f1af784c73a17266244

Have wuffsfmt put underscores in long numbers

view details

Nigel Tao

commit sha 05b790d08810b85c9fdcd59ed24529a1c8a9ebee

Refine pixel_format.bits_per_pixel return type

view details

Nigel Tao

commit sha f6f2ac4206d9d90cd13f905ac84fdcfe1b7ed801

Have std/bmp support 32 bits per pixel color

view details

push time in 17 days

push eventgoogle/wuffs

Nigel Tao

commit sha a737a75dd33b4102d3bcb564524822cf821ba523

Refactor wuffs_base__pixel_swizzler__prepare

view details

Nigel Tao

commit sha 7e4dc3bf121020cad0629b8c61ee12bdeb26f0f8

Drop obsolete "foo.min(x:etc)" comment

view details

Nigel Tao

commit sha fd196177a2c69e2338b00f0868a388f38b914b99

Add "#unsupported method" error

view details

Nigel Tao

commit sha f50e1616774c1fef510c65694dac93afe78f04e6

Add image_config_decoder interface test

view details

Nigel Tao

commit sha fc48e6518994e88693cbfedb3d55e1b6ab5c4f14

Factor out std/gif/common_consts.wuffs

view details

Nigel Tao

commit sha 4ab2de9b95469b4d0c6a3befbb4a0dbc24c2445c

Add wuffs_base__empty_{range,rect}_xx_uxx

view details

Nigel Tao

commit sha b37e55f4a5524d4287341f36b66bfde0bdd618b0

Increment coroID per-struct, not per-package

view details

Nigel Tao

commit sha 097e59853df08642c6391c7a00d4f4792216d86d

Tweak cmd/wuffsfmt/main.go

view details

Nigel Tao

commit sha 1aa501632f40701c8f86dc6b608a7297ca0ce6c1

Add gif.config_decoder (part 1 of 2) This commit (part 1) adds script/preprocess-wuffs.go but running it is deferred for part 2. Part 1 makes no change to the C code (release/c/** and test/c/**).

view details

Nigel Tao

commit sha 555e8eaaa36804bc8d6930d274941ec31b9eb464

Add gif.config_decoder (part 2 of 2) This runs script/preprocess-wuffs.go as foreshadowed by the previous commit 1aa50163 "Add gif.config_decoder (part 1 of 2)". The new gif.config_decoder type is largely a copy/paste (with minor edits) of the existing decoder type, but it can decode GIF animation metadata (e.g. frame timings) without the memory requirements of a complete GIF decoder (which can also decode pixels). As of this commit, sizeof(config_decoder) and sizeof(decoder) is 1320 and 65024, a difference of over 62 KiB. The trade-off is that binary size increased by around 1.5x or 18K for the GIF package. Future refactorings might be able to claw that back, but it might not. "wuffs genlib; ls -l gen/lib/c/*/*gif* | awk etc" output: Before: 25904 gen/lib/c/clang-5.0-dynamic/wuffs-std-gif.lo 26424 gen/lib/c/clang-5.0-static/wuffs-std-gif.o 30480 gen/lib/c/gcc-dynamic/wuffs-std-gif.lo 30064 gen/lib/c/gcc-static/wuffs-std-gif.o After: 41168 gen/lib/c/clang-5.0-dynamic/wuffs-std-gif.lo 42328 gen/lib/c/clang-5.0-static/wuffs-std-gif.o 49040 gen/lib/c/gcc-dynamic/wuffs-std-gif.lo 48304 gen/lib/c/gcc-static/wuffs-std-gif.o

view details

Nigel Tao

commit sha 3fd6aca3659f1438b5ed2a465a16a77ba57f5166

Change gif workbuf_len worst case from 1 to 0

view details

push time in 21 days

issue commentgolang/go

net/http/cookiejar: Persistent load/save cookies from/into disk/[]byte

There is a possibility, but it's not going to happen by just waiting for me to get around to it.

codenoid

comment created time in 22 days

issue commentgolang/go

proposal: image: add generic metadata support for jpeg, gif, png

some of the standard patterns (like the binary marshal/unmarshal interfaces) can't be implemented.

Well, encoding.BinaryMarshaler works with []byte, not io.Writer, so it wouldn't care about a context.Context either way.

As for Contextualizing the stdlib, I haven't really been following the discussion (other than skimming e.g. https://github.com/golang/go/issues/20280). I'd ask the golang-dev@ mailing list for advice on whether and how new stdlib API involving I/O should incorporate Contexts. For example, which is less jarring?

  • New stdlib API to ignore Contexts, despite Context being in the stdlib now.
  • New stdlib API to look inconsistent to old stdlib API, despite being in the same package.
drswork

comment created time in 24 days

issue commentgolang/go

net/http/cookiejar: Persistent load/save cookies from/into disk/[]byte

Quoting https://groups.google.com/d/msg/golang-nuts/7mf8z9bbEcY/bIjQBjePsVUJ from 2015:

Re persistent storage, there was a lot of discussion with [@vdobler] and I a couple of years ago about how to do a cookie jar with pluggable storage, but we never landed on a satisfactory design, let alone a perfect one, so what's in the standard library is not persistent.

codenoid

comment created time in 24 days

issue commentgolang/go

proposal: image: add generic metadata support for jpeg, gif, png

Metadata: map[string]encoding.BinaryUnmarshaler{
	"png/*": &kitchenSink,
},

To clarify, encoding.BinaryUnmarshaler would no longer be sufficient, but see also my previous comment suggesting some sort of interface MetadataLoader.

drswork

comment created time in a month

issue commentgolang/go

proposal: image: add generic metadata support for jpeg, gif, png

I opted not to have individual DECODE_XXX bits in the API since it seemed to add complexity but not much value, but if folks prefer the bits I can add it.

Suppose there's a new Foo image format with a Bar metadata field. It could be a little awkward (e.g. managing version skew) if the DECODE_BAR bit needed to be defined in the stdlib but "package foo" was outside of the stdlib.

Now, I consider all those "metadata", and there are definitely more than one of them per image. Having them all in a single []byte blob would be kind of weird and I'm not sure how to even do that in a way that's not more expensive and internally ugly than just decoding the damn things

...

In my code I really just want to say "gimme the !@$ metadata" and get back a metadata struct... And when I want to update the metadata I want to just stuff properly typed data into fields

Well, it's not a single []byte blob. There'd be a map entry for gamma, a map entry for creationTime, etc. Or, if you want everything, we could possibly let you write:

kitchenSink := png.Metadata{}
img, _, err := md.DecodeWithOptions(os.Stdin, &image.DecodeOptions{
	Metadata: map[string]encoding.BinaryUnmarshaler{
		"png/*": &kitchenSink,
	},
})

The user, the package main author, doesn't have to know about []byte at all, only the package png author, or if it's more modular, the package exif author too. See src/image/example/rotate.go in https://go-review.googlesource.com/c/go/+/216799 for a lack of explicit []byte anywhere, and instead the exif.Exif type indeed has properly typed fields (ex.Rotation).

there's the issue where there's recursive decoding

Interesting. More thought needed.

In any case, as I've said, I think this discussion would be more productive if we could see the equivalent of 216799's examples (my CL) in 208559's API (your CL).

drswork

comment created time in a month

issue commentgolang/go

proposal: image: add generic metadata support for jpeg, gif, png

treat metadata as an opaque io.{Reader,Writer} (why even use []byte?)

I chose []byte because it seemed simpler (my CL is really a quick sketch, I haven't tried implementing it end-to-end), and because the encoding.Binary{Marshler,Unmarshaler} interfaces already exist. I don't know of any equivalent interfaces for io.{Reader,Writer} in the stdlib.

io.ReaderFrom does exist and sort of matches syntactically, but doesn't really match "something that can initialize itself by parsing from an io.Reader" semantically.

In any case, if PNG iTXt metadata is structured so that two iTXTt chunks mean something different than one iTXT chunk (containing the concatenation), then neither []byte nor io.Reader would work. I could imagine something like interface MetadataLoader{ LoadChunk(chunk []byte) error }, but more experimentation would be needed.

drswork

comment created time in a month

issue commentgolang/go

proposal: x/net/html: add RawNode

I was concerned about security implications, but you're right that we can already render arbitrary (insecure) HTML with a CommentNode.

Let's do this: https://golang.org/cl/216800

MagicalTux

comment created time in a month

issue commentgolang/go

proposal: image: add generic metadata support for jpeg, gif, png

Copy/pasting from the https://go-review.googlesource.com/c/go/+/208559 discussion, for those here following only the issue:

A couple of (possibly combinable) other ideas to explore are:

  • Have image.DecodeWithOptions also take an arg (perhaps a map) that lets the caller say "I want EXIF, but don't care about XMP".

  • Have format-specific packages (e.g. package jpeg, package png) just treat metadata as an opaque []byte or io.Reader / io.Writer, and it's up to the caller (e.g. package main) to forward them on to metadata-specific packages (e.g. package exif, package iccp).

Discussing my suggestions should go smoother with actual code examples. There's a sketch at https://go-review.googlesource.com/c/go/+/216799 which might need some tweaking in the details, but hopefully gives the broad idea.

drswork

comment created time in a month

push eventgoogle/wuffs

Nigel Tao

commit sha 621bd9505d111001d3d79d07e25453be27c42391

Add script/draw-with-mask.go

view details

Nigel Tao

commit sha 4a83eba3f93443a0624c24df3fc01eb40da2f31a

Add WUFFS_BASE__PIXEL_BLEND__SRC_OVER

view details

Nigel Tao

commit sha 731350f8f95775396fc55aee06d59cf7b8f297cf

Optimize imageviewer's XCB_EXPOSE handling

view details

Nigel Tao

commit sha d595980f7586a0b1c539dc7e51bfecb6b0d4d6da

Let imageviewer toggle background color

view details

Nigel Tao

commit sha 86a2a6231384084691b83ef01dbb603670d51bd0

Add "#unsupported pixel swizzler option" error

view details

Nigel Tao

commit sha 47ce6ea2965e4507139e117e0120eeeec2ddbb24

Add wuffs_base__pixel_swizzler__func typedef

view details

Nigel Tao

commit sha 557c2b7998f876352c74d569f2e02f282c2e6a42

Add wuffs_base__pixel_buffer__color_u32_at

view details

Nigel Tao

commit sha 3a0b3104b07c3bd3ae57eb6be0663d68427be11a

Add wuffs_base__pixel_buffer__set_color_u32_at

view details

Nigel Tao

commit sha f0bf894ad7534cbc26d89867658e283e2c547838

Add /README.md table of contents to doc/README.md

view details

Nigel Tao

commit sha 071c97769ad88494400e808871df99e3fa559281

Have gifplayer work with uint8_t* buffers

view details

Nigel Tao

commit sha a2d6483b3b017017a8b014b9be6fee9dc327fb96

Remove example/gifplayer intermediate pixel buffer

view details

Nigel Tao

commit sha b78161715998eb0af3bea72cc6e69c5b01aaa62f

Re-organize example/gifplayer global variables

view details

Nigel Tao

commit sha 9373749b0791d93b7263e1c06291626571ab65d9

Have example/gifplayer use calloc

view details

Nigel Tao

commit sha 0e833f909a0a9018b3e39015f9eebc182e403a69

Add WUFFS_BASE__PIXEL_FORMAT__BGR_565

view details

Nigel Tao

commit sha 55aedb34f9f421a3f226a6cf6ce81136c3006ec0

Simplify gif.decoder.copy_to_image_buffer bpp calc

view details

Nigel Tao

commit sha 7527103ec4d2f861a8a14cea39dbaf8a04e955b9

Clean up image-impl.c's loop_unroll_count

view details

Nigel Tao

commit sha 67b8c79ce8807ea42b1658810c92d2f82b74746c

Fix "-Werror=conversion" build failure re BGR_565

view details

Nigel Tao

commit sha db87fa6bd18de9de563f3bb352eaabee3616916a

Changelog for WUFFS_BASE__PIXEL_BLEND__SRC_OVER

view details

push time in a month

issue commentgolang/go

compress/flate: implement partial flush

I think it's @dsnet's call about what to do re @nhooyr's latest comments.

hanwen

comment created time in a month

issue commentgolang/go

proposal: x/net/html: add RawNode

I'm not keen on using golang.org/x/net/html for server-side rendering. Use the standard library's html/template instead. Yes, the former has nodes and the latter is text based, but the latter is explicitly designed to render HTML securely.

MagicalTux

comment created time in a month

issue commentgolang/go

x/image/tiff: grayscale tiled images are not decoded correctly

I dropped some review comments on https://go-review.googlesource.com/c/image/+/211237

Sorry for the late reply.

zerkms

comment created time in a month

issue commentgolang/go

x/image/font/sfnt: implement font.Face

See also https://go-review.googlesource.com/c/image/+/170578 "font/opentype: implement Glyph and GlyphBounds" which is incomplete and abandoned, but might be educational for anyone who wants to tackle this themselves.

sbinet

comment created time in a month

issue commentgolang/go

x/image/font/sfnt: implement font.Face

I can't speak for others, but for myself, I'll repeat what I've said before: I'd like to see this happen, but I don't have any time to work on this myself.

sbinet

comment created time in a month

push eventgoogle/wuffs

Nigel Tao

commit sha c6e6dd70236a8924252678e37c26afb7bf036c97

Add Chris Palmer to CONTRIBUTORS "Google Inc." is already in AUTHORS.

view details

push time in a month

pull request commentgoogle/wuffs

Fix some typos in the documentation.

Thanks for the thorough reading of the docs!

noncombatant

comment created time in a month

push eventgoogle/wuffs

Chris Palmer

commit sha e9fdc383002a3749c9e69a06b910ccc29b0d83d7

Fix some typos in the documentation. And some minor clean-ups. There should be no semantic changes to the documentation in this patch.

view details

Nigel Tao

commit sha f1f388e1ef0e0ba01ca4b747e8d8e2643a6fd440

Merge branch 'master' into fix-typos

view details

Nigel Tao

commit sha 894782b404fab4c67a62f21f6b5f2cdb0626e667

Merge pull request #25 from noncombatant/fix-typos Fix some typos in the documentation.

view details

push time in a month

PR merged google/wuffs

Fix some typos in the documentation.

And some minor clean-ups. There should be no semantic changes to the documentation in this patch.

+39 -41

3 comments

19 changed files

noncombatant

pr closed time in a month

push eventnoncombatant/wuffs

Nigel Tao

commit sha 7d4a3a127d1729e730de9f173e2978dd65813b24

Increase cgolz4 Writer buf len for older lz4 libs

view details

Nigel Tao

commit sha fb54ba452f9248b742bda752ff72e31d21c1cc57

Add wuffs_base__make_status

view details

Nigel Tao

commit sha c50f16e58bfed7daae92280f689ad766187b97e2

Separate status and status_msg local variables

view details

Nigel Tao

commit sha e6db7f6a40dc5ad3517a341f230bc0c7b6891620

Make wuffs_base__status a struct

view details

Nigel Tao

commit sha 77a5311c4bc9ce7b0908d3b6550d937fd9786a4a

Add CHECK_STRING testlib macro

view details

Nigel Tao

commit sha 7f1572ca16289f7988f123336af8718832525164

Add CHECK_STATUS testlib macro

view details

Nigel Tao

commit sha 0319777ccef90a6b22dbade9dc37db50e1e081bb

Have tests print all of status.repr

view details

Nigel Tao

commit sha 0fa32bd09f57f159ca9853e4133d2bfa14c4a79b

Rename warnings to notes

view details

Nigel Tao

commit sha fabfff05d15ffa493da1b984052f6b7d660e458c

Have pixel_foo methods take pointer receivers

view details

Nigel Tao

commit sha 4f53f3101fb4d8682a8b2c4fce9a2de9b0ea69cd

Make wuffs_base__pixel_format a struct

view details

Nigel Tao

commit sha c4b699a1829a6ecc79be55716844e9c5c7fb7cee

Add LICENSE crc32 sanity check to build-all.sh

view details

Nigel Tao

commit sha b666ab06f7ac0d12f19e25ddef20dac1291a2660

Make wuffs_base__pixel_subsampling a struct

view details

Nigel Tao

commit sha 9e802a5f84d1b329c904d191578e7fc81a49ea28

Have wuffs_base__pixel_config__set take repr args

view details

Nigel Tao

commit sha 5bf070b187bcbd69fa273c877c83abad4dd64344

Add a wuffs_base__vtable type

view details

Nigel Tao

commit sha 4c9be03baa9ad4b47b090e9311ec330e00843741

Add interface struct declarations

view details

Nigel Tao

commit sha 989e7b1c2c13977c5d9b23ea428cedc9e0209760

Delete C++ implicit constructor for wuffs types

view details

Nigel Tao

commit sha 9e033579c81cf5c650c8a29bc11286120ab6eb22

Add interface method signatures

view details

Nigel Tao

commit sha 0c63d6e4892e8e0aa943ce17676bf6321c7b960e

Rename writeFunctionSignature modes

view details

Nigel Tao

commit sha 4323a16707d1639bbb607ea18fef7cc7d2918227

Add interface method implementations

view details

Nigel Tao

commit sha eeae6098b65da4e5be07f7b84c763846f0cb74cf

Add "do ... while(0)" around CHECK_ETC test macros

view details

push time in a month

push eventgoogle/wuffs

Nigel Tao

commit sha 47b9dcd118348376f7399e98041fd7299a2536a1

Have wbmp's restart_frame check call_sequence

view details

Nigel Tao

commit sha f5151fa1f96234c63015de2835be6e5b603a727d

Remove some obsolete basic tests

view details

Nigel Tao

commit sha 1dcb5a9283a0e5d76dbb04684a5576a84c61050e

Add wbmp.decoder.skip_frame

view details

Nigel Tao

commit sha 09ed091fe3bf6a2faf2027847b788b7156c5f8af

Rename skip / copy_n_etc to skip32 / copy_n32_etc

view details

Nigel Tao

commit sha 6ef3ffe6c87a3bcbf897fac468fb9a3f2ec64cbc

Add io_reader.skip

view details

Nigel Tao

commit sha 0d0a74c9a8873ad25855ca3d2981128b0820962c

Fix wbmp.decoder.skip_frame final call sequence

view details

Nigel Tao

commit sha 149204c14e4d729a975dc68b35dd8cab4ac96d7a

Remove wbmp seen_etc fields

view details

push time in a month

push eventgoogle/wuffs

Nigel Tao

commit sha 8562951be6d374e825f8b0167955bd200175b88b

Re-order "#ifdef __WORDSIZE" check

view details

Nigel Tao

commit sha 1a791217de297350df5dc6abdb1466504dc10413

Add wuffs_base__pixel_blend type

view details

Nigel Tao

commit sha 2be1f63806d8351f18d92fdadf66d867c0f0a77a

Remove wuffs_base__frame_config__blend

view details

Nigel Tao

commit sha 8f1fe43abf2bdefee21d3527572e781a4a7ffe2c

Add example/imageviewer

view details

Nigel Tao

commit sha 3128500e20a045256cfc90939a7bafd24ef24cb8

Add script/convert-png-to-wbmp.go

view details

Nigel Tao

commit sha b4ff1bfb7fd6b4f03145b415f68299dad464b8f9

Add std/wbmp package

view details

Nigel Tao

commit sha 2ae4a8a21c119062b629a2c6197cfcbe554e7125

Add wbmp.decoder.ack_metadata_chunk

view details

Nigel Tao

commit sha d9958e2524ce75c7e74d98ee17c624969601719d

Add wbmp.decoder base.image_decoder stubs

view details

Nigel Tao

commit sha bec5d6fa5c499863251a009639f227839db59e4c

Implement wbmp.decoder.decode_frame_config

view details

Nigel Tao

commit sha 4feb641841a07ca77c7ffa101895fbe19e636d28

Implement wbmp.decoder.decode_frame

view details

Nigel Tao

commit sha e2e51786a0d2c639c2b37a065e0bd82472a09885

Add more test/data/*.wbmp files

view details

Nigel Tao

commit sha fdd0675f51ef4066ba5505cd9e868c9747df4eae

Avoid "-Werror=conversion" build failure

view details

Nigel Tao

commit sha 67dbeb1d09db7a74fa7caf949825a45d7c9d8461

Rename wuffs_base__pixel_swizzler__copy_etc

view details

push time in a month

push eventgoogle/wuffs

Nigel Tao

commit sha ab0b2fd2aae0904ce4f36d818efb0704fe755cc9

Add test/data/github-tags.json wget -O github-tags.json https://api.github.com/repos/google/wuffs/tags

view details

Nigel Tao

commit sha 5f8d2da74f1902509cfed8c6fde3b7f8ef7200f1

Rename decode_io_writer methods to transform_io

view details

Nigel Tao

commit sha 6a90728bc95c8c4efdfedf1928217295c33db4bd

Build fix for hand-written decode_huffman_fast

view details

Nigel Tao

commit sha bb1e6a5844a00c27f0b9253e13282a6b71fa2b9f

Remove unused check.Checker argument

view details

Nigel Tao

commit sha d307fe3a42b8f496d12806c8f7158d7d73dde2c2

Remove unused "#ifdef"-related cgen strings

view details

Nigel Tao

commit sha e9897f6efb781af140e917c39242e7b74bcad5fe

Honor MaxImplements

view details

Nigel Tao

commit sha a6ee0ee39115f06feeb761d047e15a545297609e

Remove builtin.ParseFuncs "generic" argument

view details

Nigel Tao

commit sha 2fff7c782f3d0167fe5d7a51b45a7ecd3589572b

Unexport some lang/builtin GenericXxx constants

view details

Nigel Tao

commit sha 4b669a41812aad55eed537c43ad6fa3a406f4e2d

Add dead weight to C++ sizeof values

view details

Nigel Tao

commit sha e9fa8d3c28ec6c8cba53005dc6ba897b36c1b136

Add base.io_transformer

view details

Nigel Tao

commit sha 92a5bfd7ece0386e95857fb6075872197536b899

Have various structs implement base.io_transformer

view details

Nigel Tao

commit sha adf63e23b01af76d4e3fc6d2fbdb6a500d2c6433

Add base.io_transformer.workbuf_len

view details

Nigel Tao

commit sha e77c5d70015e5eb4b90deca475f240537c6a1b34

Have hasher_u32 test take a filename arg

view details

Nigel Tao

commit sha c8e3a94200b39a7bf386a2708a31c7c11d9a83ed

Add base.image_decoder

view details

push time in a month

push eventgoogle/wuffs

Nigel Tao

commit sha 5bf070b187bcbd69fa273c877c83abad4dd64344

Add a wuffs_base__vtable type

view details

Nigel Tao

commit sha 4c9be03baa9ad4b47b090e9311ec330e00843741

Add interface struct declarations

view details

Nigel Tao

commit sha 989e7b1c2c13977c5d9b23ea428cedc9e0209760

Delete C++ implicit constructor for wuffs types

view details

Nigel Tao

commit sha 9e033579c81cf5c650c8a29bc11286120ab6eb22

Add interface method signatures

view details

Nigel Tao

commit sha 0c63d6e4892e8e0aa943ce17676bf6321c7b960e

Rename writeFunctionSignature modes

view details

Nigel Tao

commit sha 4323a16707d1639bbb607ea18fef7cc7d2918227

Add interface method implementations

view details

Nigel Tao

commit sha eeae6098b65da4e5be07f7b84c763846f0cb74cf

Add "do ... while(0)" around CHECK_ETC test macros

view details

Nigel Tao

commit sha 8f5983f15124d7f075491396cefec0293011b034

Tweak ast.Struct field numbering

view details

Nigel Tao

commit sha ea54c2ea11901cd0e632baa3951de88d56e3d825

Add syntax support for implementing interfaces

view details

Nigel Tao

commit sha ea47116edc7496d728073a29a78216ffa633ef40

Implement vtables

view details

Nigel Tao

commit sha 72fcc2d94fa5b5d2643cad7559a1b4bc987363e8

Add upcast methods

view details

Nigel Tao

commit sha fd5c0143ab3a32edb7827cc75d0b9ff7c5207494

Fix testlib -Werror=pedantic failures

view details

Nigel Tao

commit sha 4fd9637e5a4c57049c736eff55fe2584bd940147

Emit stricter vtable function-pointer type

view details

Nigel Tao

commit sha b8e69fa9e78880fed1e92f5d1780745d3655375a

Fix "wuffs genlib" for wuffs_base__foo__func_ptrs

view details

push time in 2 months

push eventgoogle/wuffs

Nigel Tao

commit sha 7d4a3a127d1729e730de9f173e2978dd65813b24

Increase cgolz4 Writer buf len for older lz4 libs

view details

Nigel Tao

commit sha fb54ba452f9248b742bda752ff72e31d21c1cc57

Add wuffs_base__make_status

view details

Nigel Tao

commit sha c50f16e58bfed7daae92280f689ad766187b97e2

Separate status and status_msg local variables

view details

Nigel Tao

commit sha e6db7f6a40dc5ad3517a341f230bc0c7b6891620

Make wuffs_base__status a struct

view details

Nigel Tao

commit sha 77a5311c4bc9ce7b0908d3b6550d937fd9786a4a

Add CHECK_STRING testlib macro

view details

Nigel Tao

commit sha 7f1572ca16289f7988f123336af8718832525164

Add CHECK_STATUS testlib macro

view details

Nigel Tao

commit sha 0319777ccef90a6b22dbade9dc37db50e1e081bb

Have tests print all of status.repr

view details

Nigel Tao

commit sha 0fa32bd09f57f159ca9853e4133d2bfa14c4a79b

Rename warnings to notes

view details

Nigel Tao

commit sha fabfff05d15ffa493da1b984052f6b7d660e458c

Have pixel_foo methods take pointer receivers

view details

Nigel Tao

commit sha 4f53f3101fb4d8682a8b2c4fce9a2de9b0ea69cd

Make wuffs_base__pixel_format a struct

view details

Nigel Tao

commit sha c4b699a1829a6ecc79be55716844e9c5c7fb7cee

Add LICENSE crc32 sanity check to build-all.sh

view details

Nigel Tao

commit sha b666ab06f7ac0d12f19e25ddef20dac1291a2660

Make wuffs_base__pixel_subsampling a struct

view details

Nigel Tao

commit sha 9e802a5f84d1b329c904d191578e7fc81a49ea28

Have wuffs_base__pixel_config__set take repr args

view details

push time in 2 months

issue commentgolang/go

image/draw: optimize DrawMask when drawing a Uniform Image onto a Paletted Image

Proposed fix looks fine for whenever the Go 1.15 tree opens, after the Go 1.14 release.

The recommended process for contributing patches is documented at https://golang.org/doc/contribute.html

Or, if you're happy without formal authorship attribution, in terms of the Author field in the git log, I'm happy to send out the code review with this fix.

pjbgtnj

comment created time in 2 months

issue commentgolang/go

x/exp/shiny/screen: Weird transition when uploading scaled texture in Ubuntu and macOS

Yeah, the intent of the API isn't to select "nearest" vs "bilinear" at texture creation time (NewTexture), it's to select it at draw time (Draw and DrawOptions). Any given texture can be Draw'ed multiple times, with different DrawOptions each time.

So the fix isn't to change NewTexture. The fix is to make the Drawer methods look in the (possibly nil) *DrawOptions and set the driver-specific thing based on the (not implemented yet) DrawOptions.Scaler field.

For the x11driver, you could probably add a render.SetPictureFilter call next to the render.SetPictureTransform calls.

Supporting DrawOptions needs doing not just for the x11driver, but ideally for the other drivers too: OpenGL and Win32.

It's not going to be a trivial change, and I don't have a lot of spare time right now to provide detailed guidance. Sorry.

manu-chroma

comment created time in 2 months

push eventgoogle/wuffs

Nigel Tao

commit sha 144a6165b9bac2208ea38d7e5e1f23fc3a22c679

Fix "phising" typo

view details

Nigel Tao

commit sha 6f7e9dfa26bf3c7586146de90030dff443f4339f

Update /README.md

view details

push time in 2 months

push eventnigeltao/nigeltao.github.io

Nigel Tao

commit sha f3c98616868690d71b976ce294ccee6393bd5c42

Add "Wuffs v0.2.0 is Released" blog post

view details

push time in 2 months

push eventgoogle/wuffs

Nigel Tao

commit sha 5256cb67d95c17095778abf30fd6b67b39d1d578

Add hello-wuffs-c example Fixes #9

view details

Nigel Tao

commit sha 46b2ae3e9c631397b88c0428169b5ba580629f9f

Update doc/note/io-input-output.md for io_bind

view details

Nigel Tao

commit sha ddf246b461402d71c270d8dc6bf417ee62d4f3c4

Update doc/note/facts.md for facts vs truths

view details

Nigel Tao

commit sha 06c3c3c6156461ff637265e43f60f4168a6f6c4a

Add various glossary entries

view details

Nigel Tao

commit sha 6f175c409861c6a901234bdddfc2e2317f172b9a

Restructure the top level README.md

view details

Nigel Tao

commit sha 35dfa6869633b5d281b20a23fe7c00b2efbade45

Update doc/changelog.md

view details

Nigel Tao

commit sha d8526b96ad02cbaf3697c6ba78445a8b055886d3

Update release/c/README.md for v0.2

view details

Nigel Tao

commit sha b36c9f019f2908639b0bb4dd5b8337bb3daf3271

Update doc/changelog.md for timezones

view details

Nigel Tao

commit sha fb600fe3ce5e161936849148aa4d3ad82dfc5743

wuffs gen -version=0.2.0

view details

push time in 2 months

issue closedgoogle/wuffs

Add a "hello world outside of std" example

If anyone wants to play with Wuffs today, it's fairly safe to assume they'd want the code to live somewhere outside of the std directory. Forking and symlinks are a possibility, but they're not a very comfortable way to do things.

I propose that we add a simple example of what a working Wuffs program would be. For example, a dead simple program that you can compile to a working binary:

  • A capitalize.wuffs file, that takes a stream of bytes, and converts all lowercase a letters to A, before writing to an output stream
  • A main.c file, that uses the func/type from Wuffs above and puts it in between stdin and stdout
  • A Makefile or script that, using wuffs-c and $CC, generates the C code and compiles a working executable

In total, this should all be under 50 lines of code. Yet it would be a great starting point for those wanting to try something of their own with Wuffs.

Split from #4 - there I mentioned how it took me some digging through the Wuffs codebase to figure out how to make it work outside of the std directory.

closed time in 2 months

mvdan

created taggoogle/wuffs

tagv0.2.0

Wrangling Untrusted File Formats Safely

created time in 2 months

issue commentgoogle/wuffs

Do you use Wuffs? Tell us!

The Wuffs GIF decoder is being trialled by Skia, the 2-D graphics library used by both the Android operating system and the Chromium web browser.

nigeltao

comment created time in 2 months

issue openedgoogle/wuffs

Using wuffs-c (not wuffs) is awkward

The wuffs command line tool is used when working on Wuffs-the-Library. The wuffs-c command line tool is (implicitly) part of that, but it can also be used on its own for other code written in Wuffs-the-Language, including code that doesn't live in this repository.

Doing that works... sort of. But it's not a great experience. The hello-wuffs-c example has to declare a placeholder wuffs-base.c file (as it can't rely on the generated gen/c/wuffs-base.c file), and the C driver program has to #define WUFFS_CONFIG__MODULE__BASE.

Both of these workarounds shouldn't be necessary.

created time in 2 months

push eventgoogle/wuffs

Nigel Tao

commit sha a5dbb9160f2e44a830dd9d3ce9ea8b6cc4e06f15

Add doc/note/assertions.md

view details

Nigel Tao

commit sha e7fb1558deca5cbf5eab115ba77dcbb6f6a7c89a

Clarify wuffsfmt ignores . in paths not contents

view details

Nigel Tao

commit sha 4e58808746e60cdd93ffab7c2753bdea6565b752

Update doc/wuffs-the-language.md

view details

Nigel Tao

commit sha 61c463273bdba71204d8892152985828f67b654f

Update doc for v0.2 release date

view details

Nigel Tao

commit sha 1536b3b6ba1293fde4b3f33862b5f7caddc2a211

Add doc/note/memory-safety.md

view details

Nigel Tao

commit sha 90141ad39f4d88553e9fd03db38ffafbbbca313a

Add doc/README.md

view details

Nigel Tao

commit sha aaa30fc5a6dfc8158161fc87978c2e73da8447d3

Add doc/note/iterate-loops.md

view details

push time in 2 months

push eventgoogle/wuffs

Nigel Tao

commit sha 7e6c43e521a7a9aa7504cae3b8503487e89f39f3

Fix doc/related-work.md typo

view details

Nigel Tao

commit sha 9a5a3f636d217b42cf3b5cde93559a16f025bf99

Check that status messages start with one of @#$

view details

Nigel Tao

commit sha d1864b0faf288df58e619c10e3aad300924ee890

Add internal/README.md

view details

Nigel Tao

commit sha 355a95eeea6e50a4e883e4b7f47f8bc4f5c86f6d

Add wuffs_base__status__message

view details

Nigel Tao

commit sha 5ad98921e77337a44c6c6e19c38aae68bbbdc908

Update doc/note/facts.md

view details

Nigel Tao

commit sha 89bbe3011562da5865bb2d0d34cbd27c525cddb2

Update bounds-checking note for overflow-checking

view details

Nigel Tao

commit sha d098582cc635fa1713c6cafcbe94f479bbd70faa

Add image-decoders.md link to Skia example

view details

push time in 2 months

issue commentgolang/go

x/exp/shiny/screen: Weird transition when uploading scaled texture in Ubuntu and macOS

Yeah, this looks like "nearest neighbor" vs "bilinear" scaling. To confirm that, on Ubuntu, you could changing the "bilinear" strings to "nearest" strings at:

https://github.com/golang/exp/blob/2f50522955873285d9bf17ec91e55aec8ae82edc/shiny/driver/x11driver/screen.go#L362

There's an existing TODO to add a DrawOptions field to control this:

https://github.com/golang/exp/blob/2f50522955873285d9bf17ec91e55aec8ae82edc/shiny/screen/screen.go#L353

Unfortunately, I don't have any spare time to work on this myself. Sorry.

manu-chroma

comment created time in 2 months

issue commentgolang/go

x/image/tiff: grayscale tiled images are not decoded correctly

/cc @bsiegert

zerkms

comment created time in 2 months

push eventgoogle/wuffs

Nigel Tao

commit sha 8c8d9d8b52dd2306f12cacf4970a3728a166c377

Fix "noticable" typo

view details

Nigel Tao

commit sha 0f375517e1a9c8de466bbbc34fccaf0dabff5b51

Add 32-bit ARMv7 benchmarks

view details

Nigel Tao

commit sha 721678ed75cd006645a9b207b48d506598470273

Add doc/note/pixel-subsampling.md

view details

Nigel Tao

commit sha 3923bb9637b4c90f465e3996071a04c41238cbeb

Change pixel_subsampling from shift to divide

view details

Nigel Tao

commit sha 770b1203f42db6eaca65c08194e7051e3f262a05

Add glossay entry for effect syntax

view details

Nigel Tao

commit sha ad787cba11eb5aa8bbe37b2fb160fbbe87f5f6ee

Change ractool default codec from zlib to zstd

view details

Nigel Tao

commit sha 7b5e6146186aac5f5220dbad53fdb27d38c3915e

Update doc/note/coroutines.md for args changing

view details

Nigel Tao

commit sha d0e393e0ceb0c436ce47010f1c74b6b18cb5f67d

Update doc/std/hashers.md for timing attacks

view details

Nigel Tao

commit sha 1f8348487c158b9119d5d90e6f0da4683761ced8

Canonicalize relative markdown links

view details

Nigel Tao

commit sha 0152bd9ef4845844667b423fd9c86ba0700cc6e0

Add script/print-markdown-links.go

view details

Nigel Tao

commit sha 2f1d5cd1dd85874bf05741d343f7b866a009bcd3

Add glossary entry for methods.

view details

Nigel Tao

commit sha f32ad2869ca6b5002004f6704e0a9db3173bf781

Add doc/note/statuses.md

view details

push time in 2 months

issue commentgolang/go

proposal: image: add generic metadata support for jpeg, gif, png

Of course, if you just want to add creator metadata to images, and arguing about encoding/xml depenencies, Go stdlib release cycles and API stability is adding up to a daunting set of hurdles, then as discussed above, an alternative approach is to just start your own github.com/google/drsimage/image/png repository.

Personally, I wouldn't be too worried about abandoned code. GitHub is full of inactive projects, but that doesn't mean that they aren't useful (either directly, as fork targets, or simply for studying). Culturally, Go is also better than most other programming languages in that, if your project targets Go 1.x, it should still continue to build and run for years, without any changes.

drswork

comment created time in 3 months

issue commentgolang/go

proposal: image: add generic metadata support for jpeg, gif, png

I'm neutral on providing facilities to limit the damage caused by invalid image types. How important is that, given that dying a horrible OOMy death isn't a regression from things currently?

Well, the point isn't that OOMing on malicious input wouldn't be a regression. The point is that, if you're going to introduce basically "version 2" of the image decoding API, it'd be nice to avoid having to introduce "version 3" later if we want to fix the OOM problem, or e.g. #8055.

This is in terms of API. We could, in theory, implement this issue in one Go release, and fix OOMs in a later TBD release. But we can't change API once it's released.

drswork

comment created time in 3 months

issue commentgolang/go

proposal: image: add generic metadata support for jpeg, gif, png

Since the XML parser is part of the standard library and it doesn't seem to have much of a program footprint impact it didn't seem worth the effort to be clever.

Well, we explicitly test that the standard library's packages' dependencies don't unintentionally change:

https://golang.org/src/go/build/deps_test.go

It is possible to add dependencies between stdlib packages, but now matter how small, it would probably involve discussion and getting approval from the golang-dev@ mailing list.

I'm not saying it's necessarily a good idea, but an alternative design idea is to have Metadata be more or less a dumb map[string][]byte, and it's up to the caller, not the image/png package per se, to look in the map for the "xmp" key and, if present, pass those bytes to a xmp parser.

drswork

comment created time in 3 months

issue commentgolang/go

proposal: image: add generic metadata support for jpeg, gif, png

I dropped a bunch of low-level, tactical comments on https://go-review.googlesource.com/c/go/+/208559

A broader, more strategic comment: it's a good start, but I think that there's still a lot of work to go. I think it would help us reviewers understand it works if we could see some example programs. I don't need to see an actual working implementation, but I'd like to see how the various API pieces come together for the caller. For example:

  • decode a JPEG from stdin, and rotate according to EXIF orientation.
  • decode a JPEG from stdin, re-encode as PNG (preserving EXIF metadata) to stdout.
  • decode an image from stdin (in GIF, JPEG or PNG) and print XMP author metadata (if present).
  • feel free to add whatever itch you're personally motivated to scratch.

I'd also like to know what parts of #27830 (and its linked issues), if any, you'd like to rule in or out of scope.

drswork

comment created time in 3 months

push eventgoogle/wuffs

Nigel Tao

commit sha f4da5382915e9939bbc54bced7d87c99230ec72c

Fix broken link to decode_quirks.wuffs

view details

Nigel Tao

commit sha 03f0d84a3a3f978d3236fd42f86344308ca1fd37

Remove -Werror from "wuffs test/bench"

view details

Nigel Tao

commit sha a4f7761182b15f4865c77e7cadfc7ccb353c0003

Add glossary entry for utility types

view details

Nigel Tao

commit sha 5362ca7573654b8425d2273728f187e8d4267931

Add doc/note/pixel-formats.md

view details

Nigel Tao

commit sha 3325f730415bf4804a25b4c6b5a0f0eb9ca8cf7c

Let set_from_{slice,table}'s pixcfg arg be const

view details

Nigel Tao

commit sha 8926a85e159e755b00e4bcd75fbab6bbacb47061

Add doc/note/coroutines.md

view details

Nigel Tao

commit sha 688e947016bfac10e05a892ad0b8857409dec33b

Check that =? types are base.status

view details

Nigel Tao

commit sha 39d7ced75f0de30f56bba18f97fd7e4633fff5f6

Add doc/note/effects.md

view details

push time in 3 months

push eventgoogle/wuffs

Nigel Tao

commit sha 2a894eea2f33f375de7cc2cbd1a30c6ed19fd53b

Spin out std/gif/decode_quirks.wuffs

view details

Nigel Tao

commit sha 892e2bbde9ccc11093288ad7d0a3e54bcc2d94d9

Update partial-zero-initialization doc link

view details

Nigel Tao

commit sha 0201b8545599adead7bae253bd8059365c7c051f

Add doc/std/image-decoders.md

view details

Nigel Tao

commit sha e32d680ca4d51a16bbe51231e27f5cb89730a8df

Add doc/std/README.md

view details

Nigel Tao

commit sha fd38af62dc0a981e133e7dd03f13999651eccdb5

Add doc/std/compression-decoders.md

view details

Nigel Tao

commit sha 5342208341c135b14e6663e19dfd2f21447ff415

Add glossary entries: coroutines, effects, flicks

view details

Nigel Tao

commit sha 1174f2db82be4a25a19160c8ef77d10e9e1c73fc

Delete doc/compatibility.md

view details

push time in 3 months

push eventnigeltao/taowm

Nigel Tao

commit sha 0d0cd69bf41060bda979e37de53004cd29b301da

Add option to show battery percentage

view details

push time in 3 months

push eventgoogle/wuffs

Nigel Tao

commit sha cb718b4463ade368a2a16418944cc34a9c582887

Add glossary entry for facts

view details

Nigel Tao

commit sha 9e375309a4bf3e1a093c3655a28d7530aa0d1c0b

Add doc/wuffs-the-library.md

view details

Nigel Tao

commit sha 8ff9cdb9a67a6c5b6d4a074354e0cfd5bf98080c

Rename hasher update_u32 method

view details

Nigel Tao

commit sha a96756bd09dcf8cc6b7293e09a9154538eab6a00

Add doc/std/hashers.md

view details

Nigel Tao

commit sha b45b28daf8b22c02b070d6d3b8c37d17e63e0fd9

Rename doc/note/zero-initialization.md

view details

Nigel Tao

commit sha 73f0f28815b9d2bf22c3bda8ebd2c8a1cc3a0fa1

Elaborate doc/note/initialization.md

view details

push time in 3 months

push eventnigeltao/nigeltao.github.io

Nigel Tao

commit sha 4915064f05239469f11e32c72620b63e0c7f13c0

Add "XYZ ABC Problem" blog post

view details

Nigel Tao

commit sha 3b615531f812f8d9bdb347b53a3c50dad0061a5c

Add links to rac and wuffs projects

view details

Nigel Tao

commit sha 2720780b7244582ae789a25352c46d71984656c7

Add script/update.go

view details

Nigel Tao

commit sha f38d45e3a31054c68286be6538e011328a373692

Add feed.xml

view details

push time in 3 months

push eventgoogle/wuffs

Nigel Tao

commit sha 760277a58ce65f0b28de0f60847a3b7b60589334

Add doc/note/fuzzing.md

view details

Nigel Tao

commit sha 1609be052f4f3d6edbae491d195edfb07f312858

Add doc/note/signed-unsigned-integers.md

view details

Nigel Tao

commit sha 53cb271c4b96699ad14a8b01b3e275ccfc091dcf

Spin out a NIE Related Work document

view details

Nigel Tao

commit sha e1091ec1827818d957c656ae8c2261347c8ed682

Re-order some token.ID numbers

view details

Nigel Tao

commit sha cb7e9e0ce9404794fc27ba34b5edcd8e16ffa0dd

Add a ~mod* operator

view details

Nigel Tao

commit sha f10849d20bd4916b8a5ffe61fb15c6454e2e93c5

Add doc/note/zero-initialization.md

view details

Nigel Tao

commit sha afb76b9b505f8fa996093ef1db48c6c6426df908

Add glossary entry for dependent,refinement types

view details

Nigel Tao

commit sha a781e038bb0f3ef6282c8bf131e33346eae0e651

Update zero-initialization example

view details

Nigel Tao

commit sha f888089e9b2e99e8d8c32b012e71e7c061053abc

Factor out doc/note/bounds-checking.md

view details

Nigel Tao

commit sha 191c6bc6f28e681c1052affd72b2c048f15e8159

Add glossary entry for operator precedence

view details

push time in 3 months

push eventgoogle/wuffs

Nigel Tao

commit sha 7ce58b6179b8ceff729651607b80b805a17480cc

Add IntRange.TryFoo methods

view details

Nigel Tao

commit sha 4ce51308960961a40a3a463b2fe30e2971887368

Update doc/roadmap.md

view details

Nigel Tao

commit sha a0ace850d17925ec59a85ca5ac9bf1ce56801c00

Add doc/interval-arithmetic.md

view details

Nigel Tao

commit sha 04e8d14d40b506a26899b1a364a276c93446053e

Add doc/glossary.md

view details

Nigel Tao

commit sha 9557d89c5658b16d713d5c1afaa8ad80170da620

Add example/gifplayer/README.md

view details

push time in 4 months

issue commentgolang/go

x/image/font/gofont/gomedium: wrong shape for "l" letter

I'll poke Bigelow & Holmes again about it. If they're still busy, we (the Go team) may be able to issue our own fix to their fonts. I'll get back to you...

MichaelMonashev

comment created time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha e7d30aa3bdc3568b60eb303295b742712b15a740

Remove unnecessary "#ifdef WUFFS_IMPLEMENTATION"s

view details

Nigel Tao

commit sha 7ec252876541ec203659949450fafddc148b606e

wuffs gen -version=0.2.0-rc.3

view details

push time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha 70ac26a2c574470caa3c3f297e886954c83b2dd4

Tweak lib/interval/radial_test.go if-else ordering

view details

Nigel Tao

commit sha 607bd735b4aaf21b385ec74656d7536cc2c55ef6

Have IntRange methods return distinct *big.Int's

view details

Nigel Tao

commit sha 6a9a3abd24e9dbc8d4144c4bdf09941b5563c3c5

Let IntRange's And and Or handle negative values

view details

Nigel Tao

commit sha f3c9c4005e38b48afe9a412623e95b2c519e9c1e

Optimize lib/interval tests' opKey look-up

view details

Nigel Tao

commit sha 2ab322ee44d8fa03127ef208be2daac3a6333f01

Cache lib/interval's small bit-masks

view details

Nigel Tao

commit sha 194aaa0a7bb2d4c3d546caaeecc3b539193f7de0

Remove a few IntRange.{And,Or} allocations

view details

Nigel Tao

commit sha 3b56ace877e125932db97880a39e2bb714c6c1ba

Rename lib/interval's makeEmptyRange

view details

Nigel Tao

commit sha 648932e0fb97b800480d95d1959bbfd2df01af14

Add a lib/interval sharedEmptyRange global

view details

Nigel Tao

commit sha 960a3b76d7ded16d2b7cc6ba85c8058dd8e50854

Have lib/interval's split2Ways return emptiness

view details

Nigel Tao

commit sha d90ad9d282ab46d224e9c05e0067e3414029decc

Tweak lib/interval's split3Ways' return values

view details

Nigel Tao

commit sha 29c7b678748b6303c0c5203ec346c8dcc3d0f2b4

Rename lib/interval's split2Ways variables

view details

Nigel Tao

commit sha 8ce3d657eea61167fb4049206a7cc27de00304bc

Add IntRange.ContainsNonNegative

view details

Nigel Tao

commit sha 498b7138f3d7e1a63e8db78357ad355a43a84d0e

Remove "static const uint32_t etc" from cgen They trigger compiler warnings (-Wunused-const-variable) precisely because they're unused.

view details

Nigel Tao

commit sha 80cb158dde1b2a40902d0d47cdaf759efaa515de

wuffs gen -version=0.2.0-rc.2

view details

push time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha de60610741e01466698d047c04520be2665aa0f2

Add script/print-byte-frequencies.go

view details

Nigel Tao

commit sha 8d86f3dace115b70745d1f66486cf2ceeed9c7c3

Update doc/note/* for sub-slice syntax

view details

Nigel Tao

commit sha c8087550ec2a0ba057b794466cbf528459e959e9

Add doc/note/ranges-and-rects.md

view details

Nigel Tao

commit sha 645d17fb20a660e51ff535285f1e5e5d8e059bd6

Add doc/slices-arrays-and-tables.md link; fix typo

view details

Nigel Tao

commit sha de807374d24b7b6f783c17bfb6380adad74c962a

Have lib/interval use [m ..= n] syntax

view details

Nigel Tao

commit sha 5672e7f0a62bd3cafe0a7419ef3fa436c18271ee

Add IntRange.Unite and IntRange.Intersect

view details

push time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha 3cf06d4585a15d5a5be442f877186c7fc2b38bd4

Have cgozstd optionally use zstd 1.4's stable API

view details

Nigel Tao

commit sha e7561604331597c512ef7055fb6e3f23b1bb7847

Add axioms.md note to run `go generate`

view details

Nigel Tao

commit sha bd96dad3335c6d43f263907f81f31305c7316f34

Update ractool's numbers for zstd-with-dictionary

view details

push time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha 2fd864986631d8834de3d6aa2d863163fff9d7b3

Support cgozstd dictionaries Updates #23

view details

Nigel Tao

commit sha 7136dcdfd484214c321f36d1848216b3996f5c21

Move cgozstd dictionary init earlier, into Reset

view details

Nigel Tao

commit sha 3b466277dc29520ad93403505528ac28d416b99e

Fix "wuffs gen" null pointer dereference

view details

Nigel Tao

commit sha 2d8648ae35b34eedc9f67b9820c1f12f93a5921f

Implement raczstd dictionaries

view details

Nigel Tao

commit sha cdb60a950d5f59a8a22d24c911daae465dc7de6a

Tweak LEAVE_INTERNAL_BUFFERS_UNINITIALIZED comment

view details

Nigel Tao

commit sha 8ac2af01224109d6552f08a3ba28063e4a6d555f

Change raczlib's dictionary wire format This makes raczlib and raczstd use the same format: RAC's "common dictionary format". This is a breaking change to any existing RAC+Zlib files that use shared dictionaries. RAC+Zlib is still an experimental (unstable) file format.

view details

push time in 4 months

issue commentgolang/go

proposal: Go 2: reduce noise in return statements that contain mostly zero values

The ... in func Printf(format string, a ...interface{}) means zero or more.

jimmyfrasche

comment created time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha b92ea038897646a788abfa20de1e567692913cd8

Fix mimiclib/deflate-gzip-zlib.c build with miniz

view details

Nigel Tao

commit sha 7b0391f4a2028c63815078b0edc7c03445b10a74

Update doc/benchmarks.md for other Deflate impls

view details

push time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha 078d8a6b9cd0d127bbf1f796fb506bfcb146c0a3

Have script/bench-rust-gif print the rustc version

view details

Nigel Tao

commit sha d2b0e1c20300ea11dfc4ceccfe745aa93bb63f91

Refine script/bench-rust-gif crates.io URL

view details

Nigel Tao

commit sha 135f706b3e7f3502e21f48b81391df867198e219

Add script/bench-rust-deflate

view details

Nigel Tao

commit sha 29c23cd50568815ab034cc4482ccebf6c4278693

Add a warm-up rep to script/bench-rust-gif

view details

Nigel Tao

commit sha e442d4593caf42d0c047c7aefa4d833cca9b4ab5

Add num_want_bytes to script/bench-rust-gif

view details

Nigel Tao

commit sha b56d7c393a39d5d7e5f8be0846ff6305e57f8262

Fix script/bench-rust-deflate typo

view details

Nigel Tao

commit sha 4fc6d2af7b3d273472024fff75e7e1c0c260b31b

Avoid script/bench-go-gif variable shadowing

view details

Nigel Tao

commit sha a9576964fd8fe1cb6077fea1777546b4384d6c14

Add script/bench-go-deflate

view details

Nigel Tao

commit sha e384a0b38783533a00343fb1f4896063b8aa0be7

Have script/bench-go-gif load files less often

view details

push time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha 903088b537de4375f3cacedf7c8aa9d02597b253

Update doc/binary-size.md

view details

push time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha 7d761404293a8b97ee56e1fc48b80bf93700c44c

Have "wuffs genlib" take a -ccompilers arg

view details

push time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha c84351364061a23fca20f6078f93ca60c9b5a64e

Fix doc/benchmarks.md typos

view details

Nigel Tao

commit sha b44c823fdfea3a55459e674451d92d84329df72f

Factor out lang/check/axioms.md

view details

push time in 4 months

issue commentgolang/go

x/image: CCITT reader fails to return last row of data

First, thank you @nigeltao and others

The others, @hhrutter and @bsiegert deserve more thanks than me.

decoding but not "packing" the last line

This is possibly related to https://go-review.googlesource.com/c/image/+/198547 which @hhrutter has coincidentally also sent out recently.

As for this issue (and CL 200162), perhaps @hhrutter and @bsiegert can take a first look at this and how it relates to CL 198547. I'm busier than usual with other things for the next few weeks.

dchang-dchang

comment created time in 4 months

IssuesEvent

issue commentgolang/go

x/net/html: add offset & line tracking, case preservation

I'll re-open the issue to consider adding a Tokenizer.RawTagAttr or similar method.

Nonetheless, I would recommend something like <vg type="main.DemoLine" bind="Num:i" etc> instead of <main:DemoLine :Num="i" etc>, for a few reasons:

  1. This will play better with other HTML tools that assume that, as per the spec, HTML is case-insensitive. For example, pretty-printers might canonicalize tags and attribute keys to lower case.
  2. The main.DemoLine looks more Go-like (a package-qualified type name) than main:DemoLine.
  3. You don't have to worry about any HTML/XML special-case weirdness re svg:foobar tags if your Go package happens to be called svg.
bradleypeabody

comment created time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha 66348290e31a44ae8b5fbcd7039ac955477cf0a9

Update doc/benchmarks.md

view details

push time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha df66093473ae7935a665242e494b88f897c6a5b0

Have bench-rust-gif process multiple images

view details

Nigel Tao

commit sha ffdce5ef180083a58052a0194262c8f1624e602f

Have bench-rust-gif process animated / RGBA images

view details

push time in 4 months

issue openedGeal/gif.rs

Does not build with nom v5

$ git clone https://github.com/Geal/gif.rs.git
$ cd gif.rs/
$ cargo build
    Updating crates.io index
    Updating git repository `https://github.com/Geal/nom`
   Compiling semver-parser v0.7.0
   Compiling arrayvec v0.4.12
   Compiling ryu v1.0.2
   Compiling nodrop v0.1.14
   Compiling memchr v2.2.1
   Compiling version_check v0.1.5
   Compiling cfg-if v0.1.10
   Compiling static_assertions v0.3.4
   Compiling byteorder v0.3.13
   Compiling semver v0.9.0
   Compiling nom v5.0.1 (https://github.com/Geal/nom#85170646)
   Compiling rustc_version v0.2.3
   Compiling lexical-core v0.4.6
   Compiling gif v0.0.1 (/dev/shm/tmp/gif.rs)
error[E0432]: unresolved imports `nom::ErrorKind`, `nom::be_u8`, `nom::le_u8`, `nom::le_u16`
 --> src/parser.rs:1:44
  |
1 | use nom::{HexDisplay,Offset,Needed,IResult,ErrorKind,be_u8,le_u8,le_u16};
  |                                            ^^^^^^^^^ ^^^^^ ^^^^^ ^^^^^^ no `le_u16` in the root
  |                                            |         |     |
  |                                            |         |     no `le_u8` in the root
  |                                            |         no `be_u8` in the root
  |                                            no `ErrorKind` in the root

error: cannot find macro `length_bytes!` in this scope
   --> src/parser.rs:140:17
    |
140 |     data:       length_bytes!(be_u8)          >>
    |                 ^^^^^^^^^^^^ help: you could try the macro: `length_data`
etcetera
etcetera
etcetera

created time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha ff7bffcffc5be57931f2b7c090364c743fc4c8bb

Calculate deflate Huffman table worst case size

view details

Nigel Tao

commit sha 098cad9c9555fbba7de7e36e75908e90daa8acfd

Add links to more wire format examples

view details

Nigel Tao

commit sha dc1218fec0496df3a465ec55a83a92da25e6c1b0

Spell std/deflate's "literal/length" consistently

view details

Nigel Tao

commit sha 948c6fb239ed4ea8602a3fc290e30b50c067d214

Avoid gcc's -Werror=format-truncation for snprintf

view details

Nigel Tao

commit sha a24a1d427e892d9a83b5df53ac126906768cdbc8

Avoid gcc's -Werror=array-bounds for memset

view details

push time in 4 months

push eventgoogle/wuffs

Nigel Tao

commit sha 69e9428c1b2d75be898a7a6938a16d54f4ee9f11

Fix pdst_mark for hand-written decode_huffman_fast

view details

Nigel Tao

commit sha 18c6e5716a92822b2426b211bb19e8c6955fd97c

Fix deflate-etc.deflate.commentary.txt typo

view details

Nigel Tao

commit sha d501daddc3949b25d71c2ffd8439aa097f3d2014

Rename `[i..j]` to `[i ..= j]`, the same as Rust

view details

Nigel Tao

commit sha 5d29c7df61add8b55829949d895d5891d25202da

Remove unused token.IDDollar

view details

Nigel Tao

commit sha 7872a8893b0cd334e6b6dae049f0b76d145669aa

Rename while:label to while.label

view details

Nigel Tao

commit sha a9ea63ff06a3a2997e6d811b3a75cb4ceb26d69b

Rename `[i:j]` to `[i .. j]`, the same as Rust

view details

Nigel Tao

commit sha 5b4e3647c0f7536e6802929c7927c269a5e6800d

Reformat `x:y` as `x: y`

view details

Nigel Tao

commit sha 1520a1f3459bcfd3622fea533b456fc691b6cd71

Rename `x T` to `x: T` for fields and func args

view details

Nigel Tao

commit sha a21ad25bb8b87fae3c3762b635937785546564c3

Rename `x T` to `x: T` for vars

view details

Nigel Tao

commit sha 358ad7f91827f684c028de41bda7ffe31a13494f

Add an explicit deflate-etc.deflate.decompressed

view details

push time in 4 months

created taggoogle/wuffs

tagv0.1.0

Wrangling Untrusted File Formats Safely

created time in 4 months

issue openedfacebook/zstd

ZSTD_dm_etc should be ZSTD_dct_etc

Some files (e.g. in lib/zlib.h) have comments like "the prefix is interpreted as raw content (ZSTD_dm_rawContent)", which refers to names like:

  • ZSTD_dm_auto
  • ZSTD_dm_rawContent
  • ZSTD_dm_fullDict

These should presumably be updated to:

  • ZSTD_dct_auto
  • ZSTD_dct_rawContent
  • ZSTD_dct_fullDict
$ grep --files-with-matches --recursive ZSTD_dm_ .
./tests/fuzzer.c
./doc/zstd_manual.html
./lib/zstd.h
./lib/compress/zstd_compress.c

I'd make the PR myself, except that I'm not sure whether doc/zstd_manual.html should be manually updated or generated by some tool. In any case, it should be easy for y'all to fix.

created time in 5 months

issue commentgolang/go

x/net/html: add offset & line tracking, case preservation

Certain attributes, e.g. vg-if-"expr" trigger corresponding functionality in the emitted Go code.

Should the second - in vg-if-"expr" be an = instead? If so, the "expr" part doesn't get lower-cased:

package main

import (
    "fmt"
    "log"
    "os"

    "golang.org/x/net/html"
)

func main() {
    f, err := os.Open("vugufmt/testdata/ok/root.vugu")
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()

    for z := html.NewTokenizer(f); ; {
        tt := z.Next()
        if tt == html.ErrorToken {
            break 
        }   
        if (tt != html.StartTagToken) && (tt != html.SelfClosingTagToken) {
            continue
        }
        tok := z.Token()
        for _, a := range tok.Attr {
            if a.Key != "vg-if" {
                continue
            }
            fmt.Printf("%s vg-if %q\n", tok.Data, a.Val)
        }
    }
}

This program prints:

div vg-if "data.isLoading"
div vg-if "len(data.bpi.BPI) > 0"

and e.g. the "bpi" and "BPI" cases are preserved. The html package lower-cases the attribute keys but not the values.

the attribute names correspond to Go struct field names

Can you give an example, then, if vg-if doesn't actually have a case-sensitivity problem, AFAICT?

I looked at https://www.vugu.org/doc/files/markup but didn't see anything that needed case-sensitive attribute keys. I'm assuming that the click in <button @click="etc"> isn't case-sensitive.

bradleypeabody

comment created time in 5 months

issue commentfacebook/zstd

API for user-specified Dictionary_ID's confusing (or missing?)

You would be better off providing a raw dictionary with no statistics.

Or, if the cooked dictionary format is Magic_Number, Dictionary_ID, Entropy_Tables, Content, can I hard-code the bytes-on-the-wire encoding of a "no statistics" Entropy_Tables? My mental model is to somehow set up the same initial state as if there was no dictionary, but I don't know if that's the right mental model.

If I can hard-code those bytes, what would those bytes look like?

nigeltao

comment created time in 5 months

issue commentfacebook/zstd

API for user-specified Dictionary_ID's confusing (or missing?)

if I seeded some fake statistics with 256 samples, each one byte long

That's not a good direction, really. You would be better off providing a raw dictionary with no statistics.

OK, does "no statistics" mean calling ZDICT_finalizeDictionary with the nbSamples argument set to zero, or does it mean not calling ZDICT_finalizeDictionary at all?

I'm happy with not providing statistics, but as per the OP, I would still like to provide a dictionary ID, to indicate in the zstd data that it is expecting a (raw) dictionary. While the spec suggests to me that this is possible, there's no API to combine a raw dictionary with a dictionary ID, right?

Or are you suggesting that if I'm using zstd raw dictionaries, I should also not specify a dictionary ID, and rely on the decompressor to know out-of-band which dictionary to use?

The idea of a "universal dictionary" has been a thing, and compared to "nothing" it was certainly a progress. But as we are improving our understanding of this technique, it becomes clear that this was just an intermediate stage, leaving a lot of capabilities on the table.

Sure, format-specific cooked dictionaries are better than format-agnostic raw dictionaries, but raw dictionaries are still better than no dictionaries, and I'd still like to get some data points on raw dictionaries.

nigeltao

comment created time in 5 months

issue commentfacebook/zstd

API for user-specified Dictionary_ID's confusing (or missing?)

2 important factors, source size, and compression levels.

Ah, so the dictionary statistics are only used when compressing, and not when decompressing?

At low compression levels, algorithm will basically "trust" the statistics, in order to save time,

Would it be feasible to have an API switch (either through an explicit function call or to have some special-cased "nonsense" statistics values) to say "don't trust the statistics; spend the time"?

Higher compression levels are able to make more intelligent choices

How much is "high"? For example, is "compression level 9" considered high?? Would 9 (or 12, or 19, or whatever) be "disastrous" if I seeded some fake statistics with 256 samples, each one byte long (i.e. a uniform distribution), just to satisfy the ZDICT_finalizeDictionary API?

It's really possible to use the output of zstd --train as a raw dictionary

OK. The cooked (non-raw) dictionary format is: Magic_Number, Dictionary_ID, Entropy_Tables, Content. The first two terms are tiny (4+4 bytes). How big are the Entropy_Tables?

Even so, I'd still like to be able to work with raw dictionaries (e.g. use other dictionary-generating programs like Brotli's dictionary_generator.cc, which obviously doesn't speak the zstd cooked dictionary format). Even though zstd --train has a --block-size parameter, I'd still like to be able to experiment with different dictionary selection algorithms and implementations. That Brotli program comes with three different algorithms (what they call engines): https://github.com/google/brotli/blob/c435f066751ef83aa4194445085a70ad9d193704/research/dictionary_generator.cc#L102

You're right that I could just use zstd's format as a slightly longer raw dictionary (with a funny prefix), but then suppose I'd also like to work with a new compression format in the future (for argument's sake, let's call it middle-out), and it also has its own cooked dictionary format. I then can't produce the one dictionary.dat file that is both zstd's cooked format and middle-out's cooked format. On the other hand, it'd be nice if "raw dictionary" was a common denominator that worked with every codec.

nigeltao

comment created time in 5 months

push eventgoogle/wuffs

Nigel Tao

commit sha efb18036cbc3b1cd0f69d6eddf2dcc63d424e3fa

Add gif-small-frame-interlaced.gif test

view details

Nigel Tao

commit sha aaebb17a0183b750954e082700da5ad18de6c75e

Tweak how std/gif does interlace row replication

view details

Nigel Tao

commit sha 9089d375788906f7ba9e7ca3576587e7238f6771

Clip std/gif replicated rows to the frame rect Updates https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=17802

view details

Nigel Tao

commit sha 745b5c41e0759f0d0684232ac0e1fa1fb173bc06

Give test/c/std/gif error messages more context

view details

Nigel Tao

commit sha 97bcdded0d2ef3ee9955d9684906183b73d3cae4

Remove obsolete TODO

view details

Nigel Tao

commit sha 1c1ce9b19f19ddbf1c75793c730e8a1519f3d6fc

Drop the unused ref and deref operators

view details

Nigel Tao

commit sha b0f18dacbd3cc446673d213acfb7a9317153a9ce

Remove deprecated wuffs_base__null_etc functions

view details

Nigel Tao

commit sha 4080840928c0b05a80cda0d14ac2e2615f679f1a

wuffs gen -version=0.2.0-rc.1

view details

Nigel Tao

commit sha df839b183f0c0a2c250920281f542daa085fbe41

Build fix for hand-written decode_huffman_fast

view details

push time in 5 months

issue commentfacebook/zstd

API for user-specified Dictionary_ID's confusing (or missing?)

Otherwise, statistics are off, and are at best not useful, at worst detrimental.

Detrimental as in "incorrect output" or merely "worse performance"? If it's just "worse performance", roughly speaking, would you expect "a little worse", "a lot worse", or "it depends"?

If I wanted to read more about how the statistics are used, do you have any suggestions (other than the obvious "read the source code")? For example, should I just start with http://fastcompression.blogspot.com/2014/01/fse-decoding-how-it-works.html and go forward through your blog?

I thought it was about the RAC format, which I believed implies that there are some data blocks to pack together in this format ?

Sorry for not being clearer. Yes, this is RAC, but RAC is not tied to zstd, there's also RAC+zlib, RAC+lz4, RAC+brotli, etc. In general, I like to try multiple compression algorithms, and pick which one works best for my users' particular corpus.

The workflow separates dictionary generation from RAC file generation. They are literally two different programs. This is partly because one is C++ and one is Go, so it's not trivial to put it all in one monolithic program. But also because the dictionary can be re-used across compression codecs (zlib, zstd, etc): if I want to compare multiple different codecs on the same source file, I still only have to create the dictionary once (in theory).

The dictionary program is, for me, often Brotli's dictionary_generator.cc that I mentioned in the OP, not zstd --train. Importantly, even if I wanted to use zstd --train with --block-size=#, the dictionary generator needs to output a dictionary.dat file that is codec-agnostic (to work with the multiple compression formats): a raw dictionary, not anything zstd-specific, so no samples.

The RAC file generation program takes the dictionary file, and the source file as stdin. It treats the input as a stream, not something that is seekable or rewindable. Thus, when encoding the first chunk, it would like to use the dictionary, but it doesn't yet know the upcoming N-1 chunks to make a realistic sample.

That's why I said "I don't really have a set of sources to make a representative sample". Of course, it's all software, and software is infinitely malleable, but I was hoping to avoid making zstd a special case in my workflow, such as "use this dictionary generation program for RAC+zstd, use this other one for RAC+anything_else".

Should I file a separate issue to track stabilizing zdict.h API?

I filed #1816.

nigeltao

comment created time in 5 months

issue openedfacebook/zstd

No stable zdict.h API for rolling your own dictionaries

lib/dictBuilder/zdict.h has the ZDICT_trainFromBuffer function to train a dictionary. There's other API (e.g. ZDICT_params_t, ZDICT_finalizeDictionary) that lets you create a dictionary from your own training algorithm. But that other API is not stable: it's behind an #ifdef ZDICT_STATIC_LINKING_ONLY guard, and my (Go) program does not static-link zstd.

See #1776 for motivation.

I know that "just bless the existing unstable API as stable" can be easier said than done. I just want a zstd-the-project issue to point people to if I'm asked why my program doesn't support zstd-the-format dictionaries yet.

created time in 5 months

issue commentfacebook/zstd

API for user-specified Dictionary_ID's confusing (or missing?)

Also, ZDICT_finalizeDictionary() is not stabilized yet (it's behind #ifdef ZDICT_STATIC_LINKING_ONLY), which makes it harder for my (Go) program to use it.

Should I file a separate issue to track stabilizing zdict.h API?

nigeltao

comment created time in 5 months

issue commentfacebook/zstd

API for user-specified Dictionary_ID's confusing (or missing?)

ZDICT_finalizeDictionary() requires accessing a few target samples to compress, on top of 3rd-party generated content, so that statistics information present in the header are somewhat meaningful.

OK, can I just say: here's one (fake) sample, the single byte "\x00"? Or should I give a few more (one-byte) samples, or should I give multi-byte samples? What's the implication if the statistics are 'wrong'? At the time I'm encoding the dictionary into my file, I don't really have a set of sources to make a representative sample, let alone a complete set.

nigeltao

comment created time in 5 months

issue closedgolang/go

x/net/html: add offset & line tracking, case preservation

After looking at https://github.com/golang/go/issues/31312 closer and trying some things out, I wanted to open a new issue with a more specific and succinct proposal:

Features

At a high level, the desired features are:

  1. Line and column tracking (actually offset tracking, see below)
  2. A way to get at the original element and attribute name text before lower casing.

Motivation

The motivation for 1 is cases where a caller needs to know the context of a token or node so this can be reported to the user. Indicating the position of an error, or where in the source HTML a particular element comes from, etc.

The motivation for 2 is for situations where the caller simply needs to know what was originally there before lower-casing. It could be useful in error reporting to show the original element name for example. (The case for me specifically is I'm doing code generation based on HTML and so allowing the user to use mixed case in a tag name means they can specify the exact name of a Go struct with CamelCase - the documents in question are mostly standard HTML but of course these elements are treated differently.)

Observations

  • Tracking the byte offset of each token and node is simple and unambiguous.

  • Line and column information can be derived from this offset, but raises other questions like the definition of a line ending, how do tabs impact the reported column position, multi-byte characters, etc.

  • Thus these concerns would seem to be better handled outside the parser - if the parser reports the byte offset into the original input, that's enough.

  • Providing an option to keep the original case of an element in the Data field seems, in retrospect, dangerous. I can't prove that this won't cause things to break (the parser does many weird and wonderful things internally to handle special cases with different tags - for which I have a new found respect after diving deeper into the source :) ). A new field seems much more sensible.

  • Both tracking offset information and preserving the original text require fields to be added to the appropriate structs (see below). The performance impact of them seems negligible. I thus opted to not implement them as ParserOptions, as it would provide little benefit to disable these features - the fields would still be there just unpopulated.

Changes

Here's a summary of how this could work (how it works in the prototype):

  • Add OrigData string; Offset int to Token. OrigData is the same as data but before lower casing. (If they are the same they point to the exact same bytes.)

  • Add OrigData string; Offset int to Node, same behavior (fields are just copied from Token to Node).

  • Add OrigKey to Attribute. Original Key before lower casing. (In retrospect Attribute probably could have an Offset too, I might add that.)

  • And basically just the book keeping to fill out those fields.

  • Separately, a LineCounter struct is added which is wraps an io.Reader and records the positions of every line ending and provides a call to go from an offset to line number and line start offset. Callers can use this to reconstruct "line:col" given a byte offset from Node.Offset, etc. Notice that the question of how to deal with tabs or multi-byte characters becomes the caller's responsibility; this is intentional. (I didn't bother to implement alternate line endings, it assumes \n for now, but this could be done easily.) I'm not sure if it is appropriate to have a LineCounter inside the HTML package, it could just as well be somewhere else.

Working Prototype

The working code with these modifications is here: https://github.com/vugu/html .

The main commit that touches the internal stuff is: https://github.com/vugu/html/commit/da33d265c2a8fa501665a122efbf1441162a6b3b

Some basic tests are in there, I'm sure more could be done. (I also haven't looked at code that writes out HTML to see how the originally-cased data can be used there too or if this would be an option or what.) I also understand if some of this needs to be split up, this proposal is a few different things lumped together.

I believe these features could be generally useful and so I wanted to see if this is something that could potentially end up being merged, and if so discuss what's required to get to that point.

closed time in 5 months

bradleypeabody

issue commentgolang/go

x/net/html: add offset & line tracking, case preservation

I'm going to close this issue. Re lower-casing keys, I'll repeat that HTML5 is specified to be case-insensitive, and in any case, it sounds like you can work around it. If you can't, feel free to re-open this issue with further thoughts.

bradleypeabody

comment created time in 5 months

issue commentgolang/go

x/net/html: add offset & line tracking, case preservation

is offset guaranteed to always have the correct value... it should be documented

You're right. I sent out https://go-review.googlesource.com/c/net/+/198357

bradleypeabody

comment created time in 5 months

push eventgoogle/wuffs

Nigel Tao

commit sha 51fadc4db761699d64a03ca6e5e4e0de83544456

Add script/bench-history.sh

view details

push time in 5 months

push eventgoogle/wuffs

Nigel Tao

commit sha 851b7561d9a3b4af6e6ffbdb61f510892e8b1d1a

Fix dirty rect for interlaced GIFs

view details

Nigel Tao

commit sha 2819fb8e3f5e32ea113d69b83df109a6f0e0e13b

Rename some built-in func's arg from x to a

view details

Nigel Tao

commit sha 59b2ddf37fbc3eeaa0dc7b3bb858e7e376cf4aef

Clamp std/gif dirty rect to the image height

view details

Nigel Tao

commit sha 293a633e8af2305119b8e145e1a19590178fa0c3

Clip std/gif output to the image rect

view details

push time in 5 months

push eventgoogle/wuffs

Nigel Tao

commit sha 498e1f407d40a40a2fa1c84d8b34af334573099d

Factor out deflate.decoder.add_history

view details

Nigel Tao

commit sha 18763813d5f2b2ff864f3ae2ef25b58e89768027

Support zlib dictionaries

view details

Nigel Tao

commit sha b4808596aceff0dd96b88943e07aab15226abc29

Add a ractool -tmpdir flag

view details

Nigel Tao

commit sha a9b9a82524ddd078efc49da84d04535d20aa88be

Have concurrent rac reading require an io.ReaderAt

view details

Nigel Tao

commit sha 15670546bb81de0934fd7d1067163d3e7788a597

Let std/deflate decode a degenerate H-D table

view details

push time in 5 months

more