profile
viewpoint
Ophir LOJKINE lovasoa @auto-grid Paris, France ophir.dev Senior Software engineer @auto-grid, previously worked @Qwant

lovasoa/bad_json_parsers 338

Exposing problems in json parsers of several programming languages.

gap-decoder/gapdecoder 55

Google Arts And Culture Downloader. Python script to download high-resolution images from google arts & culture.

lovasoa/bloomfilter 22

Simplistic (but fast) java implementation of a bloom filter.

j6s/ophir.php 21

PHP script that converts ODT to HTML

al34034/Qwant-Gnome-shell-integration 7

A Gnome-shell extension to be able to search Qwant directly from the overview

lovasoa/bin2png 2

Embed binary data inside an HTML file in an efficient way.

Fantasya/Bornes-Ecossaises 1

TA - Shotten Toten

liufan2312/tgs-salt-identification 1

Code for TGS salt identification competition on kaggle

lovasoa/anylang 1

Type text in any alphabet. Anylang is a javascript library that converts a phonetic transcription of a text in a language to a text written in the alphabet of the target language. Currently works with hebrew (with vowels) and russian.

issue commentimage-rs/jpeg-decoder

Error: "use of unset quantization table"

Hello, I cannot reproduce the error with the given file (desktop.jpg, sha1: 58902cedf28ca0fed0171e1f18c527346254f354). The file is decoded without errors by jpeg-decoder (tested on e2d3aa2da35333fa8d049ddfa87bb5926ed1d0d9). @ArturKovacs, are you sure you joined the right file ?

ArturKovacs

comment created time in 29 minutes

Pull request review commentimage-rs/image

make the jpeg encoder work with non-memory and non RGB8 images

 fn encode_coefficient(coefficient: i32) -> (u8, u16) {     (num_bits, val) } -fn rgb_to_ycbcr(r: u8, g: u8, b: u8) -> (u8, u8, u8) {-    let r = f32::from(r);-    let g = f32::from(g);-    let b = f32::from(b);+#[inline]+fn rgb_to_ycbcr<P: Pixel>(pixel: P) -> (u8, u8, u8) {+    use num_traits::{cast::ToPrimitive, bounds::Bounded};+    let [r, g, b] = pixel.to_rgb().0;+    let max: f32 = P::Subpixel::max_value().to_f32().unwrap();+    let r: f32 = r.to_f32().unwrap();+    let g: f32 = g.to_f32().unwrap();+    let b: f32 = b.to_f32().unwrap(); -    let y = 0.299f32 * r + 0.587f32 * g + 0.114f32 * b;-    let cb = -0.1687f32 * r - 0.3313f32 * g + 0.5f32 * b + 128f32;-    let cr = 0.5f32 * r - 0.4187f32 * g - 0.0813f32 * b + 128f32;+    let y = 65.481 / max * r + 128.553 / max * g + 24.933 / max * b;+    let cb = -37.797 / max * r - 74.203 / max * g + 112.0 / max * b + 128.;+    let cr = 112. / max * r - 93.786 / max * g - 18.214 / max * b + 128.;      (y as u8, cb as u8, cr as u8) } -fn value_at(s: &[u8], index: usize) -> u8 {-    if index < s.len() {-        s[index]++/// Returns the pixel at (x,y) if (x,y) is in the image,+/// otherwise the closest pixel in the image+#[inline]+fn pixel_at_or_near<I: GenericImageView>(source: &I, x: u32, y: u32) -> I::Pixel {+    if source.in_bounds(x, y) {+        unsafe { source.unsafe_get_pixel(x, y) }

GenericImageView::get_pixel should probably return an option by default, but this is another discussion. The way it works right now, it forces duplicated bound checks.

lovasoa

comment created time in a day

Pull request review commentimage-rs/image

make the jpeg encoder work with non-memory and non RGB8 images

 fn encode_coefficient(coefficient: i32) -> (u8, u16) {     (num_bits, val) } -fn rgb_to_ycbcr(r: u8, g: u8, b: u8) -> (u8, u8, u8) {-    let r = f32::from(r);-    let g = f32::from(g);-    let b = f32::from(b);+#[inline]+fn rgb_to_ycbcr<P: Pixel>(pixel: P) -> (u8, u8, u8) {+    use num_traits::{cast::ToPrimitive, bounds::Bounded};+    let [r, g, b] = pixel.to_rgb().0;+    let max: f32 = P::Subpixel::max_value().to_f32().unwrap();+    let r: f32 = r.to_f32().unwrap();+    let g: f32 = g.to_f32().unwrap();+    let b: f32 = b.to_f32().unwrap(); -    let y = 0.299f32 * r + 0.587f32 * g + 0.114f32 * b;-    let cb = -0.1687f32 * r - 0.3313f32 * g + 0.5f32 * b + 128f32;-    let cr = 0.5f32 * r - 0.4187f32 * g - 0.0813f32 * b + 128f32;+    let y = 65.481 / max * r + 128.553 / max * g + 24.933 / max * b;+    let cb = -37.797 / max * r - 74.203 / max * g + 112.0 / max * b + 128.;+    let cr = 112. / max * r - 93.786 / max * g - 18.214 / max * b + 128.;      (y as u8, cb as u8, cr as u8) } -fn value_at(s: &[u8], index: usize) -> u8 {-    if index < s.len() {-        s[index]++/// Returns the pixel at (x,y) if (x,y) is in the image,+/// otherwise the closest pixel in the image+#[inline]+fn pixel_at_or_near<I: GenericImageView>(source: &I, x: u32, y: u32) -> I::Pixel {+    if source.in_bounds(x, y) {+        unsafe { source.unsafe_get_pixel(x, y) }

Yes I removed the redundant bound checks because there was a ~8% performance difference. This function is called for every pixel in the image.

lovasoa

comment created time in a day

pull request commentimage-rs/image

make the jpeg encoder work with non-memory and non RGB8 images

Benchmarks

On my cpu:

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              12
On-line CPU(s) list: 0-11
Thread(s) per core:  2
Core(s) per socket:  6
Socket(s):           1
NUMA node(s):        1
Vendor ID:           AuthenticAMD
CPU family:          23
Model:               1
Model name:          AMD Ryzen 5 1600X Six-Core Processor
Stepping:            1
CPU MHz:             2127.710
CPU max MHz:         3600.0000
CPU min MHz:         2200.0000
BogoMIPS:            7185.56
Virtualization:      AMD-V
L1d cache:           32K
L1i cache:           64K
L2 cache:            512K
L3 cache:            8192K
NUMA node0 CPU(s):   0-11
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme ssbd vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca

Before

on 4431cd4baaac978a61d3597e067fe75a15f6632d

encode-jpeg/zero-L8-rawvec/64                                                                             
                        time:   [31.390 us 31.447 us 31.504 us]
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) high mild
  3 (3.00%) high severe
encode-jpeg/zero-L8-bufvec/64                                                                             
                        time:   [32.099 us 32.159 us 32.228 us]
Found 9 outliers among 100 measurements (9.00%)
  4 (4.00%) low mild
  1 (1.00%) high mild
  4 (4.00%) high severe
encode-jpeg/zero-L8-file/64                                                                             
                        time:   [33.505 us 33.597 us 33.710 us]
Found 15 outliers among 100 measurements (15.00%)
  4 (4.00%) low mild
  5 (5.00%) high mild
  6 (6.00%) high severe
encode-jpeg/zero-L8-rawvec/128                                                                            
                        time:   [116.40 us 116.64 us 116.88 us]
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild
  3 (3.00%) high severe
encode-jpeg/zero-L8-bufvec/128                                                                            
                        time:   [117.57 us 117.77 us 117.96 us]
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild
  2 (2.00%) high severe
encode-jpeg/zero-L8-file/128                                                                            
                        time:   [119.12 us 119.38 us 119.70 us]
Found 14 outliers among 100 measurements (14.00%)
  2 (2.00%) low mild
  7 (7.00%) high mild
  5 (5.00%) high severe
encode-jpeg/zero-L8-rawvec/256                                                                            
                        time:   [449.41 us 450.26 us 451.11 us]
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high severe
encode-jpeg/zero-L8-bufvec/256                                                                            
                        time:   [457.15 us 457.76 us 458.41 us]
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe
encode-jpeg/zero-L8-file/256                                                                            
                        time:   [460.44 us 462.09 us 464.31 us]
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild
  3 (3.00%) high severe
encode-jpeg/zero-Rgb8-rawvec/64                                                                            
                        time:   [95.339 us 95.475 us 95.603 us]
Found 10 outliers among 100 measurements (10.00%)
  3 (3.00%) low mild
  4 (4.00%) high mild
  3 (3.00%) high severe
encode-jpeg/zero-Rgb8-bufvec/64                                                                            
                        time:   [94.596 us 94.996 us 95.492 us]
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe
encode-jpeg/zero-Rgb8-file/64                                                                            
                        time:   [97.204 us 97.706 us 98.459 us]
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe
encode-jpeg/zero-Rgb8-rawvec/128                                                                            
                        time:   [370.11 us 370.57 us 371.06 us]
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe
encode-jpeg/zero-Rgb8-bufvec/128                                                                            
                        time:   [366.98 us 368.72 us 371.08 us]
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) high mild
  3 (3.00%) high severe
encode-jpeg/zero-Rgb8-file/128                                                                            
                        time:   [375.87 us 378.00 us 380.48 us]
Found 11 outliers among 100 measurements (11.00%)
  2 (2.00%) low mild
  4 (4.00%) high mild
  5 (5.00%) high severe
Benchmarking encode-jpeg/zero-Rgb8-rawvec/256: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.3s or reduce sample count to 50.
encode-jpeg/zero-Rgb8-rawvec/256                                                                             
                        time:   [1.4511 ms 1.4555 ms 1.4599 ms]
Found 10 outliers among 100 measurements (10.00%)
  1 (1.00%) low severe
  7 (7.00%) low mild
  2 (2.00%) high mild
Benchmarking encode-jpeg/zero-Rgb8-bufvec/256: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.6s or reduce sample count to 50.
encode-jpeg/zero-Rgb8-bufvec/256                                                                             
                        time:   [1.5323 ms 1.5573 ms 1.5881 ms]
Found 11 outliers among 100 measurements (11.00%)
  3 (3.00%) high mild
  8 (8.00%) high severe
Benchmarking encode-jpeg/zero-Rgb8-file/256: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.8s or reduce sample count to 50.
encode-jpeg/zero-Rgb8-file/256                                                                             
                        time:   [1.4905 ms 1.4980 ms 1.5063 ms]
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild
encode-jpeg/zero-Rgba8-rawvec/64                                                                            
                        time:   [104.54 us 109.02 us 113.77 us]
Found 12 outliers among 100 measurements (12.00%)
  2 (2.00%) high mild
  10 (10.00%) high severe
encode-jpeg/zero-Rgba8-bufvec/64                                                                            
                        time:   [118.06 us 124.43 us 131.13 us]
Found 8 outliers among 100 measurements (8.00%)
  8 (8.00%) high mild
encode-jpeg/zero-Rgba8-file/64                                                                            
                        time:   [101.21 us 102.03 us 103.11 us]
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) high mild
  10 (10.00%) high severe
encode-jpeg/zero-Rgba8-rawvec/128                                                                            
                        time:   [378.70 us 381.09 us 383.99 us]
Found 10 outliers among 100 measurements (10.00%)
  6 (6.00%) high mild
  4 (4.00%) high severe
encode-jpeg/zero-Rgba8-bufvec/128                                                                            
                        time:   [370.06 us 372.15 us 374.26 us]
Found 8 outliers among 100 measurements (8.00%)
  6 (6.00%) high mild
  2 (2.00%) high severe
encode-jpeg/zero-Rgba8-file/128                                                                            
                        time:   [377.04 us 378.91 us 381.40 us]
Found 7 outliers among 100 measurements (7.00%)
  5 (5.00%) high mild
  2 (2.00%) high severe
Benchmarking encode-jpeg/zero-Rgba8-rawvec/256: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.5s or reduce sample count to 50.
encode-jpeg/zero-Rgba8-rawvec/256                                                                             
                        time:   [1.5200 ms 1.5577 ms 1.6055 ms]
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) high mild
  4 (4.00%) high severe
Benchmarking encode-jpeg/zero-Rgba8-bufvec/256: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.4s or reduce sample count to 50.
encode-jpeg/zero-Rgba8-bufvec/256                                                                             
                        time:   [1.5117 ms 1.5305 ms 1.5546 ms]
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe
Benchmarking encode-jpeg/zero-Rgba8-file/256: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 8.3s or reduce sample count to 50.
encode-jpeg/zero-Rgba8-file/256                                                                             
                        time:   [1.4771 ms 1.4824 ms 1.4890 ms]
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) high mild
  4 (4.00%) high severe

After

encode-jpeg/zero-L8-rawvec/64                                                                             
                        time:   [32.250 us 32.345 us 32.460 us]
                        change: [+2.8912% +3.4885% +4.1245%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe
encode-jpeg/zero-L8-bufvec/64                                                                             
                        time:   [33.168 us 33.199 us 33.230 us]
                        change: [+2.8140% +3.1396% +3.4628%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild
  2 (2.00%) high severe
encode-jpeg/zero-L8-file/64                                                                             
                        time:   [36.548 us 36.766 us 36.980 us]
                        change: [+6.6991% +7.7092% +8.6236%] (p = 0.00 < 0.05)
                        Performance has regressed.
encode-jpeg/zero-L8-rawvec/128                                                                            
                        time:   [118.65 us 118.87 us 119.07 us]
                        change: [+1.3631% +1.7711% +2.1733%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe
encode-jpeg/zero-L8-bufvec/128                                                                            
                        time:   [123.35 us 123.50 us 123.65 us]
                        change: [+4.6422% +4.9876% +5.3330%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) low mild
  3 (3.00%) high mild
  2 (2.00%) high severe
encode-jpeg/zero-L8-file/128                                                                            
                        time:   [129.23 us 129.63 us 130.08 us]
                        change: [+8.1535% +8.9830% +9.9615%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) high mild
  4 (4.00%) high severe
encode-jpeg/zero-L8-rawvec/256                                                                            
                        time:   [470.89 us 472.24 us 473.88 us]
                        change: [+6.6203% +7.5684% +8.4745%] (p = 0.00 < 0.05)
                        Performance has regressed.
encode-jpeg/zero-L8-bufvec/256                                                                            
                        time:   [490.09 us 497.58 us 507.00 us]
                        change: [+5.1081% +6.2937% +7.4150%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe
encode-jpeg/zero-L8-file/256                                                                            
                        time:   [507.33 us 508.98 us 511.08 us]
                        change: [+9.2884% +9.7416% +10.212%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild
  2 (2.00%) high severe
encode-jpeg/zero-Rgb8-rawvec/64                                                                            
                        time:   [93.697 us 93.794 us 93.899 us]
                        change: [-1.7432% -1.3657% -0.9370%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe
encode-jpeg/zero-Rgb8-bufvec/64                                                                            
                        time:   [97.970 us 98.171 us 98.395 us]
                        change: [+3.4660% +3.9653% +4.4112%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe
encode-jpeg/zero-Rgb8-file/64                                                                            
                        time:   [100.31 us 100.56 us 100.80 us]
                        change: [+2.7879% +3.2725% +3.7512%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) high mild
  3 (3.00%) high severe
encode-jpeg/zero-Rgb8-rawvec/128                                                                            
                        time:   [370.29 us 370.82 us 371.34 us]
                        change: [-0.8643% -0.4696% -0.0861%] (p = 0.02 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  3 (3.00%) high severe
encode-jpeg/zero-Rgb8-bufvec/128                                                                            
                        time:   [378.21 us 378.77 us 379.26 us]
                        change: [+2.2794% +2.8110% +3.3725%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe
encode-jpeg/zero-Rgb8-file/128                                                                            
                        time:   [383.68 us 385.28 us 387.35 us]
                        change: [+1.6194% +2.1835% +2.7804%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) high mild
  3 (3.00%) high severe
Benchmarking encode-jpeg/zero-Rgb8-rawvec/256: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.4s or reduce sample count to 50.
encode-jpeg/zero-Rgb8-rawvec/256                                                                             
                        time:   [1.4670 ms 1.4688 ms 1.4709 ms]
                        change: [+0.5738% +0.9442% +1.3944%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 11 outliers among 100 measurements (11.00%)
  2 (2.00%) low severe
  1 (1.00%) low mild
  2 (2.00%) high mild
  6 (6.00%) high severe
Benchmarking encode-jpeg/zero-Rgb8-bufvec/256: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.7s or reduce sample count to 50.
encode-jpeg/zero-Rgb8-bufvec/256                                                                             
                        time:   [1.5076 ms 1.5097 ms 1.5117 ms]
                        change: [-12.982% -9.5031% -6.2152%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe
Benchmarking encode-jpeg/zero-Rgb8-file/256: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.8s or reduce sample count to 50.
encode-jpeg/zero-Rgb8-file/256                                                                             
                        time:   [1.5313 ms 1.5358 ms 1.5414 ms]
                        change: [+2.1698% +2.6765% +3.1755%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 10 outliers among 100 measurements (10.00%)
  2 (2.00%) low mild
  1 (1.00%) high mild
  7 (7.00%) high severe
encode-jpeg/zero-Rgba8-rawvec/64                                                                            
                        time:   [93.758 us 94.018 us 94.276 us]
                        change: [-11.178% -8.7590% -6.4863%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe
encode-jpeg/zero-Rgba8-bufvec/64                                                                            
                        time:   [95.874 us 96.001 us 96.118 us]
                        change: [-18.727% -15.155% -11.520%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) high mild
  3 (3.00%) high severe
encode-jpeg/zero-Rgba8-file/64                                                                            
                        time:   [98.376 us 98.630 us 99.003 us]
                        change: [-9.6122% -6.6357% -4.1019%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high severe
encode-jpeg/zero-Rgba8-rawvec/128                                                                            
                        time:   [361.06 us 361.56 us 362.02 us]
                        change: [-7.4191% -6.3237% -5.3349%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe
encode-jpeg/zero-Rgba8-bufvec/128                                                                            
                        time:   [374.14 us 374.79 us 375.38 us]
                        change: [+1.2540% +1.7652% +2.2694%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
  7 (7.00%) low mild
  2 (2.00%) high mild
  4 (4.00%) high severe
encode-jpeg/zero-Rgba8-file/128                                                                            
                        time:   [384.31 us 385.24 us 386.40 us]
                        change: [-0.5983% +0.5438% +1.5385%] (p = 0.34 > 0.05)
                        No change in performance detected.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe
Benchmarking encode-jpeg/zero-Rgba8-rawvec/256: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.3s or reduce sample count to 50.
encode-jpeg/zero-Rgba8-rawvec/256                                                                             
                        time:   [1.4501 ms 1.4513 ms 1.4527 ms]
                        change: [-8.3603% -6.2064% -4.3348%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) low mild
  2 (2.00%) high mild
  3 (3.00%) high severe
Benchmarking encode-jpeg/zero-Rgba8-bufvec/256: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.6s or reduce sample count to 50.
encode-jpeg/zero-Rgba8-bufvec/256                                                                             
                        time:   [1.4921 ms 1.4934 ms 1.4947 ms]
                        change: [-5.4143% -3.4958% -1.7837%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low severe
  2 (2.00%) low mild
  2 (2.00%) high mild
  2 (2.00%) high severe
Benchmarking encode-jpeg/zero-Rgba8-file/256: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.7s or reduce sample count to 50.
encode-jpeg/zero-Rgba8-file/256                                                                             
                        time:   [1.5168 ms 1.5185 ms 1.5203 ms]
                        change: [+2.3887% +2.8248% +3.2429%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) high mild
  4 (4.00%) high severe

Concusion

Overall, the performance seems slightly better for color images and slightly worse for greyscale ones. Looking at the code, it looks like there are performance optimizations that could be made, but let's maybe keep that for another PR.

lovasoa

comment created time in a day

push eventlovasoa/image

ophir

commit sha 6706ce52009075ee7ea8076587085ce123bbc41a

Fix compilation with older rust versions

view details

ophir

commit sha 11d3bd6b29aa49b98406126a70ed7a692affbcd9

Small performance improvements

view details

push time in a day

pull request commentimage-rs/image

make the jpeg encoder work with non-memory and non RGB8 images

Hi @HeroicKatora !

In dezoomify-rs, I handle missing image parts by writing black pixels in them. But this is up to the application developer to choose what to do. Anyway, I really don't think it's excessive to ask for image pixels to be infaillibly accessible when encoding an image. What do you think about the PR overall?

lovasoa

comment created time in a day

PR opened image-rs/image

make the jpeg encoder work with non-memory and non RGB8 images

Change the JPEG encoder to work on a GenericImageView instead of a byte slice

This change allows users to encode jpeg images from other sources than memory, thus finally allowing the encoding of images that do not fit in memory.

This also allows users to make the encoder work in a streaming fashion. (see #1219)

This also finally allows encoding images from other pixel formats than 8-bit RGB. The conversion to 8bit YCbCr (used in JPEG) is done on the fly during encoding.

I need this change in dezoomify-rs.

I license past and future contributions under the dual MIT/Apache-2.0 license, allowing licensees to chose either at their option.

+151 -113

0 comment

3 changed files

pr created time in a day

create barnchlovasoa/image

branch : generic-jpeg-encoder

created branch time in a day

push eventlovasoa/dezoomify

ophir

commit sha 5a86e1b883a451527d7102c3d2c02024481c9471

Remove a dead link from the tests

view details

push time in 2 days

issue closedlovasoa/dezoomify-rs

dezoomify-rs crashes when it goes out of available memory

Forgive me @lovasoa, but I've a problem. I want to dezoomify the cadastral map of Capergnanica. Following your instructions, I got the tiles.yaml file; I put it in the same folder as the .exe file; I started the .exe and it appeared on a black background: "Enter a URL or a path to a tiles.yaml file: ..."; I pasted the tiles.yaml file and started. On the verge of finishing and creating the .jpg file, dezoomify-rs unexpectedly stopped with THIS SCREEN.

Errore

What does it mean, please? Point out that:

  • I've Windows 7;
  • I've a free space of 54 GB;
  • dezoomify-rs worked great with other huge maps of the Milan Archive.

Thank you in advance.

closed time in 2 days

Arekkusu1998

issue commentlovasoa/dezoomify-rs

dezoomify-rs crashes when it goes out of available memory

Thank you everyone for beta-testing. dezoomify-rs v2.0.0 is now published. I can finally proudly close this issue.

Arekkusu1998

comment created time in 2 days

push eventsql-js/sql.js

ophir

commit sha 5c40349eff606e4a99b7b184e8b914f32e0026d8

v1.3.0

view details

push time in 2 days

created tagsql-js/sql.js

tagv1.3.0

A javascript library to run SQLite on the web.

created time in 2 days

push eventlovasoa/dezoomify-rs

ophir

commit sha 4f230905cba6b4bbe43c330c79525ec145263af7

v2.0.0

view details

ophir

commit sha e09ddd95ca3083e287a996ae94b4fddffe0c18cb

Merge branch 'master' of github.com:lovasoa/dezoomify-rs

view details

push time in 2 days

created taglovasoa/dezoomify-rs

tagv2.0.0

Zoomable image downloader for Google Arts & Culture, Zoomify, IIIF, and others

created time in 2 days

push eventsql-js/sql.js

Stephan Seidt

commit sha fc3264822bbad7c1038511005cb3210401f6be07

Add documentation for enabling extensions (#399)

view details

push time in 2 days

PR merged sql-js/sql.js

Add documentation for enabling extensions

I thought this might be a welcome change if someone searches for JSON or FTS on the landing page they’ll find this and get to recompiling instead of requesting it to be included. Worked flawlessly here, I’m impressed, thank you! And feel free to close if you think otherwise, of course.

+14 -0

1 comment

1 changed file

seidtgeist

pr closed time in 2 days

pull request commentsql-js/sql.js

Add documentation for enabling extensions

Great, thank you for contributing !

seidtgeist

comment created time in 2 days

push eventlovasoa/dezoomify

ophir

commit sha 9463e310f0aafd73c3be27d919383073011fd4ba

More helpful debug messages

view details

ophir

commit sha f5e89cf3900d4f016d14fb17db9bdac1db1d5779

Merge branch 'master' of github.com:lovasoa/dezoomify

view details

push time in 2 days

push eventlovasoa/dezoomify-extension

Ophir LOJKINE

commit sha cfe5ac676fe2b16c49b61f28e0431b63207de8a2

Update README.md

view details

push time in 2 days

push eventlovasoa/dezoomify-extension

Ophir LOJKINE

commit sha 94089ee5a9b56ae87ae9b9996232f26739a7af17

Add a link to enlarge the animated tutorial

view details

push time in 2 days

push eventlovasoa/dezoomify-extension

Ophir LOJKINE

commit sha cfa55aeb0ab75f035b191066cd77b4b158afc16c

Update README.md

view details

push time in 2 days

push eventlovasoa/dezoomify-extension

Ophir LOJKINE

commit sha e01d4f9b57dfab3cdc7fa015efc1c1998b71c80c

Add download icons on top

view details

push time in 2 days

push eventlovasoa/dezoomify-extension

Ophir LOJKINE

commit sha 352358d92677026b6fac8a3d253ecdb62cefca3e

Set theme jekyll-theme-leap-day

view details

push time in 2 days

GollumEvent

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha 98756fa9a64f791cf5e5186da5a97687ad8857b0

Avoid executing CI on non-code changes

view details

push time in 2 days

GollumEvent

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha 5fafa77616bcffdc0fc4ee1e2781e5ae504daea7

Update README.md

view details

push time in 2 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha 5aaf3f55f63468ad6dba5d1e9c4d082175792f6c

Update website

view details

push time in 2 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha d7c7e2b82455d725f65509661d1b07c364640e1c

Create the dev builds only when the code changes

view details

push time in 2 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha 52ee7924644d40a2c644ed7f01c0874303084a48

Create FUNDING.yml

view details

push time in 2 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha 1b2a6e14059e27cce66274b339e8434bfacb216e

Set theme jekyll-theme-leap-day

view details

push time in 2 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha 89d9b5d1e7c2700d4c1e7c6d87c0c76547b1d397

Set theme jekyll-theme-cayman

view details

push time in 2 days

push eventsql-js/sql.js

ophir

commit sha 751f16483c41ae324b01789b2d8021e76819a845

Update emscripten to 1.39.14

view details

push time in 2 days

issue commentsql-js/sql.js

Error in webpack build

Hello, I'm going to close this old issue, let me know if I should reopen it.

If you need to use sql.js with webpack, you can follow this example: https://github.com/sql-js/react-sqljs-demo

jlanio

comment created time in 2 days

push eventsql-js/sql.js

ophir

commit sha e9fa304bc28176e31cc51eaad3b846c1c8272595

Update SQLite to 3.32.0

view details

push time in 2 days

issue commentadamboche/python-marshmallow-union

Ambiguous serialization

You can avoid the ambiguity by defining strict fields, that only accept elements of the expected type. Here is an example:


def checkType(value, typ, many=False, err: Type[Exception] = TypeError):
    if many:
        for v in value:
            checkType(v, typ)
    if not isinstance(value, typ):
        raise err(f"Expected a value of type {typ.__name__}, got {repr(value)}")


class StrictStr(marshmallow.fields.String):
    def _serialize(self, value, attr, obj, **kwargs):
        checkType(value, str, many=kwargs.get("many", False))
        return super(StrictStr, self)._serialize(value, attr, obj, **kwargs)

    def _deserialize(self, value, attr, obj, **kwargs):
        checkType(value, str, many=kwargs.get("many", False), err=marshmallow.ValidationError)
        return super(StrictStr, self)._deserialize(value, attr, obj, **kwargs)


class StrictInt(marshmallow.fields.Integer):
    def _serialize(self, value, attr, obj, **kwargs):
        checkType(value, int, many=kwargs.get("many", False))
        return super(StrictInt, self)._serialize(value, attr, obj, **kwargs)

    def _deserialize(self, value, attr, obj, **kwargs):
        checkType(value, int, many=kwargs.get("many", False), err=marshmallow.ValidationError)
        return super(StrictInt, self)._deserialize(value, attr, obj, **kwargs)

asmodehn

comment created time in 2 days

issue commentlovasoa/marshmallow_dataclass

Union dump not working

This issue is caused by a bug in marshmallow-union. I opened a pull request to fix it: https://github.com/adamboche/python-marshmallow-union/pull/33

elvinlemmens

comment created time in 2 days

push eventlovasoa/python-marshmallow-union

ophir

commit sha 78c654cf9957ce5f1dd70c1c79c3bf6a0e26f0df

Fix invalid serialization for lists of unions Fixes #32

view details

push time in 2 days

issue openedadamboche/python-marshmallow-union

_serialize ignores the given value

The _serialize method of Union calls the serialize method (instead of _serialize) on the underlying fields. This means that it ignores the given value parameter, and may try to serialize a completely different value.

See: https://github.com/adamboche/python-marshmallow-union/blob/master/src/marshmallow_union/init.py#L66-L68

created time in 2 days

issue commentsql-js/sql.js

Support Sqlite cipher

You can either decrypt them first, and then serve the clear versions to the clients, or compile a custom build of sql.js.

shahramSo

comment created time in 2 days

issue commentsql-js/sql.js

Support Sqlite cipher

You can use the source code in this repository to compile sqlite with extensions, but this is not offcially supported by us.

shahramSo

comment created time in 2 days

created taglovasoa/marshmallow_dataclass

tagv7.6.0

Automatic generation of marshmallow schemas from dataclasses.

created time in 2 days

push eventlovasoa/marshmallow_dataclass

ophir

commit sha ffd9da8a2147aa10a681ba4c9a98fb1255111991

Add more tests for custom list field schemas

view details

ophir

commit sha 49a02118785c4492135c300930f08c4f3941b586

v7.6.0

view details

push time in 2 days

issue openedlovasoa/whitebophir

Inconsistent scrolling behavior between browsers

Mouse scrolling does not scroll at the same speed in all browsers, and in most cases, it is probably much slower than what the user wants.

created time in 3 days

pull request commentsql-js/sql.js

Add IDBFS support

The size increase will be mostly due to closure compiler's absence. Flushing caches may work, but I worry it would race between flush -> export, but then again I think we do all of that on one event loop tick so it might be okay if that's the case.

The code is completely synchronous, there is no race possible.

Kegsay

comment created time in 3 days

issue closedlovasoa/react-contenteditable

Fire Focus and Blur Out of the box

Hello. I wonder if I could trigger some functions and the ContentEditable components become focused or blur out of the box. I Mean I wanna call a function and after that the elements can get Input directly.

closed time in 3 days

theMasix

issue commentlovasoa/react-contenteditable

Fire Focus and Blur Out of the box

Hello, Yes, you can do that using refs

theMasix

comment created time in 3 days

pull request commentsql-js/sql.js

Add IDBFS support

Also, looking at the generated assets, it looks like the changes in this pull request roughly double the size of sql-wasm.js. I understand that we need to load the huge wasm file afterwards anyway, but sql-wasm.js is likely to end up on the critical path of the final web application...

Kegsay

comment created time in 3 days

pull request commentsql-js/sql.js

Add IDBFS support

It is a great feature ! However, I'm not sure we want to re-expose the full emscripten filesystem API, and change the interface of the Database constructor. Could we more simply flush caches instead of closing the database when calling export ? This would let you use it with indexeddb, and benefit all the existing users of Database.export.

Kegsay

comment created time in 3 days

issue commentlovasoa/dezoomify-rs

Input/Output Error: Wrong data size

Thank you again for reporting this bug early, and preventing it from making it to the final release !

macmacleannz

comment created time in 3 days

issue commentlovasoa/dezoomify-rs

Input/Output Error: Wrong data size

Here is the latest pre-release, where this bug is fixed.

macmacleannz

comment created time in 3 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha ad0a943d003aad45b591bf4d5c741f6eef064690

Factor pixel stuffing logic into a single function

view details

Ophir LOJKINE

commit sha 0c089f6febb03195b9dfb5020aa528f1d3214410

Fix #47

view details

Ophir LOJKINE

commit sha 0cc1fde7a524c896c20e9cd12be0bc63d788ee91

Fix warning in benchmark compilation

view details

push time in 3 days

issue closedlovasoa/dezoomify-rs

Input/Output Error: Wrong data size

Using Dev version https://github.com/lovasoa/dezoomify-rs/releases/tag/build-25e9c8d18aa0196c59d5e2050bd6fb3443b9fcd5 processed 100+ images with no problem.

For six or so images get the following error

image

on the following URL's

dezoomify-rs -l  http://www.cartomundi.fr/site/cmfiles/Cartes/SP8/d_fc1020.jpg/dzc_output.xml Egypt_d_fc1020.png

dezoomify-rs -l  http://www.cartomundi.fr/site/cmfiles/Cartes/SP8/d_fc1021.jpg/dzc_output.xml Egypt_d_fc1021.png

dezoomify-rs -l  http://www.cartomundi.fr/site/cmfiles/Cartes/SP8/d_fc1053.jpg/dzc_output.xml Egypt_d_fc1053.png

dezoomify-rs -l  http://www.cartomundi.fr/site/cmfiles/Cartes/SP8/d_fc1054.jpg/dzc_output.xml Egypt_d_fc1054.png

dezoomify-rs -l  http://www.cartomundi.fr/site/cmfiles/Cartes/SP8/d_fc1055.jpg/dzc_output.xml Egypt_d_fc1055.png

I get a corrupted PNG file. It works OK in Dezoomify so I can process manually but am interested in what causes the above error

closed time in 3 days

macmacleannz

issue commentlovasoa/dezoomify-rs

Input/Output Error: Wrong data size

Hello, and thank you for testing the beta and reporting bugs. This is highly appreciated. I am going to investigate this.

macmacleannz

comment created time in 4 days

issue commentlovasoa/dezoomify-extension

Error 429 Too Many Requests when trying dezoomify on The Peasant Wedding, Google Arts & Culture

Hello! This issue is a duplicate of https://github.com/lovasoa/dezoomify/issues/337

stefanaaltenibm

comment created time in 4 days

issue closedlovasoa/dezoomify-extension

Error 429 Too Many Requests when trying dezoomify on The Peasant Wedding, Google Arts & Culture

Hi,

I have installed the dezoomify add-on (v0.4.2) in Firefox (v76.0.1).

I am trying to use dezoomify with Google Arts & Culture to download the hi-res version of The Peasant Wedding by Pieter Bruegel at https://artsandculture.google.com/asset/peasant-wedding/hgGvote2WI8P3w?hl=en-GB&ms=%7B%22x%22%3A0.5%2C%22y%22%3A0.5%2C%22B%22%3A9.523707976632611%2C%22z%22%3A9.523707976632611%2C%22size%22%3A%7B%22width%22%3A1.843923339843752%2C%22height%22%3A1.4031991525423744%7D%7D.

I receive this error: "Error: Unable to fetch https://artsandculture.google.com/asset/peasant-wedding/hgGvote2WI8P3w The server responded: fopen(https://artsandculture.google.com/asset/peasant-wedding/hgGvote2WI8P3w): failed to open stream: HTTP request failed! HTTP/1.0 429 Too Many Requests (https://ophir.alwaysdata.net/dezoomify/zoommanager.js:405)"

Can you please help?

Many thanks.

Stefan

closed time in 4 days

stefanaaltenibm

issue commentlovasoa/whitebophir

Design template?

Well, all the user interface code is in the client-data folder, you can do whatever you want with it. Remember that WBO is published under the AGPL, so you have to make the source code of your customization public and available under the same license. The easiest way to do so is to fork this repository on github, and make your changes in your own fork.

fallenartist

comment created time in 5 days

issue commentlovasoa/whitebophir

Going straight to public board?

Hello, No, it is not possible. But you can configure a proxy in front of WBO to allow accessing only the public board if you want to.

fallenartist

comment created time in 5 days

issue commentlovasoa/dezoomify-rs

dezoomify-rs crashes when it goes out of available memory

I updated the link

Arekkusu1998

comment created time in 5 days

issue commentlovasoa/whitebophir

Start up errors

how do I access it?

You should ask your hosting provider, not me 😄

fallenartist

comment created time in 5 days

issue commentlovasoa/whitebophir

Start up errors

It lLooks like you are trying to listen to a port you don't have the right to listen to.

fallenartist

comment created time in 5 days

issue closedlovasoa/whitebophir

Start up errors

Hi, I'm trying to install a whiteboard on my server. I'm getting the following error:

[warsztatmiejski@s42]:<~/domains/makerspace.silesia.pl/public_nodejs/whitebophir>$ PORT=5001 npm start

> whitebophir@1.5.0 start /usr/home/warsztatmiejski/domains/makerspace.silesia.pl/public_nodejs/whitebophir
> node ./server/server.js

/usr/home/warsztatmiejski/domains/makerspace.silesia.pl/public_nodejs/whitebophir/server/sockets.js:42
	async function joinBoard(name) {
	      ^^^^^^^^

SyntaxError: Unexpected token function
    at createScript (vm.js:56:10)
    at Object.runInThisContext (vm.js:97:10)
    at Module._compile (module.js:549:28)
    at Object.Module._extensions..js (module.js:586:10)
    at Module.load (module.js:494:32)
    at tryModuleLoad (module.js:453:12)
    at Function.Module._load (module.js:445:3)
    at Module.require (module.js:504:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/usr/home/warsztatmiejski/domains/makerspace.silesia.pl/public_nodejs/whitebophir/server/server.js:2:14)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! whitebophir@1.5.0 start: `node ./server/server.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the whitebophir@1.5.0 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/warsztatmiejski/.npm/_logs/2020-05-20T11_13_59_795Z-debug.log

I'm not really familiar with node.js (sorry) – is this an easy fix? Thanks.

closed time in 5 days

fallenartist

issue commentlovasoa/whitebophir

Start up errors

Hello ! It looks like you are using a very old version of node.js. whitebophir requires at least node v8.0.

You have two possibilities :

  • use the provided docker container instead (see the README for more information)
  • update node.js (how to do so depends on your operating system)
fallenartist

comment created time in 5 days

delete tag lovasoa/dezoomify-rs

delete tag : refs/heads/master

delete time in 5 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha 25e9c8d18aa0196c59d5e2050bd6fb3443b9fcd5

Update fast-dev-builds.yml

view details

push time in 5 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha 50f3a7c97bd52fc9f76362839de4661e212cb2bb

Update fast-dev-builds.yml

view details

push time in 5 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha 9dfadb18c9aa022fcfba56d0befd49e0afeeb7c6

fix release name conflict

view details

push time in 5 days

issue commentlovasoa/dezoomify-rs

Unable to parse DZI file with a Byte Order Mark

Here is the right link : https://github.com/lovasoa/dezoomify-rs/releases/tag/refs%2Fheads%2Fmaster

I didn't realize the release wasn't public.

macmacleannz

comment created time in 5 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha fe86621b4175fbb643a87fd63659614d41efd03f

Make dev releases public

view details

push time in 5 days

pull request commentlovasoa/marshmallow_dataclass

Allow to map a custom Field class to a Type

Thank you for your contribution, @YBadiss and @CedricCabessa !

CedricCabessa

comment created time in 5 days

issue commentlovasoa/dezoomify-rs

dezoomify-rs crashes when it goes out of available memory

I made a few more improvements , and you can download the latest build here : https://github.com/lovasoa/dezoomify-rs/releases/tag/untagged-7390da2b90bad1bc3e9a

If no one reports any regression, then I'll probably release it this week.

Arekkusu1998

comment created time in 5 days

issue commentlovasoa/dezoomify-rs

Unable to parse DZI file with a Byte Order Mark

You can download the latest build of dezoomify-rs with the fix here: https://github.com/lovasoa/dezoomify-rs/releases/tag/untagged-7390da2b90bad1bc3e9a

macmacleannz

comment created time in 5 days

push eventlovasoa/marshmallow_dataclass

Cédric Cabessa

commit sha 3c7ab2463c0405e59608f724311817e866307973

Allow to map a custom Field class to a Type (#85) * Allow to map a custom Field class to a Type This patch allow us to map a custom implementation of a field to map to a generic type. eg: ``` class BaseSchema(marshmallow.Schema): TYPE_MAPPING = { **marshmallow.Schema.TYPE_MAPPING typing.List: MyList } @marshmallow_dataclass.dataclass(base_schema=BaseSchema) class Sample: my_custom: CustomType ``` `foo` will be a `MyList` field and not a `marshmallow.fields.List` Credits goes to @ybadiss to have developed this patch Fixes #66 * Use schema.TYPE_MAPPING to override generic types Co-authored-by: Yacine Badiss <yacine.badiss@ledger.fr>

view details

push time in 5 days

PR merged lovasoa/marshmallow_dataclass

Allow to map a custom Field class to a Type

This patch allow us to map a custom implementation of a field to map to a generic type. eg:

def dataclass(*args, **kwargs):
    kwargs["generic_types"] = kwargs.get("generic_types", {typing.List: MyList})
    return marshmallow_dataclass.dataclass(*args, **kwargs)

@dataclass
class MyClass:
  foo: List[int]

foo will be a MyList field and not a marshmallow.fields.List

Credits goes to @ybadiss to have developed this patch

Fixes #66

+31 -4

3 comments

2 changed files

CedricCabessa

pr closed time in 5 days

pull request commentlovasoa/marshmallow_dataclass

Allow to map a custom Field class to a Type

Great ! Let's just rename generic_type_mapping to type_mapping (since it does not contain only generics), and we'll merge.

CedricCabessa

comment created time in 5 days

Pull request review commentlovasoa/marshmallow_dataclass

Allow to map a custom Field class to a Type

 def field_for_schema(     origin = typing_inspect.get_origin(typ)     if origin:         arguments = typing_inspect.get_args(typ, True)+        # Override base_schema.TYPE_MAPPING to change the class used for generic types below+        generic_type_mapping = base_schema.TYPE_MAPPING if base_schema else {}+         if origin in (list, List):             child_type = field_for_schema(arguments[0], base_schema=base_schema)-            return marshmallow.fields.List(child_type, **metadata)+            list_type = generic_type_mapping.get(List, marshmallow.fields.List)+            return list_type(child_type, **metadata)         if origin in (tuple, Tuple):             children = tuple(                 field_for_schema(arg, base_schema=base_schema) for arg in arguments             )-            return marshmallow.fields.Tuple(children, **metadata)+            tuple_type = generic_type_mapping.get(Tuple, marshmallow.fields.Tuple)+            return tuple_type(children, **metadata)         elif origin in (dict, Dict):-            return marshmallow.fields.Dict(+            dict_type = generic_type_mapping.get(Dict, marshmallow.fields.Dict)
            dict_type = type_mapping.get(Dict, marshmallow.fields.Dict)
CedricCabessa

comment created time in 5 days

Pull request review commentlovasoa/marshmallow_dataclass

Allow to map a custom Field class to a Type

 def field_for_schema(     origin = typing_inspect.get_origin(typ)     if origin:         arguments = typing_inspect.get_args(typ, True)+        # Override base_schema.TYPE_MAPPING to change the class used for generic types below+        generic_type_mapping = base_schema.TYPE_MAPPING if base_schema else {}+         if origin in (list, List):             child_type = field_for_schema(arguments[0], base_schema=base_schema)-            return marshmallow.fields.List(child_type, **metadata)+            list_type = generic_type_mapping.get(List, marshmallow.fields.List)+            return list_type(child_type, **metadata)         if origin in (tuple, Tuple):             children = tuple(                 field_for_schema(arg, base_schema=base_schema) for arg in arguments             )-            return marshmallow.fields.Tuple(children, **metadata)+            tuple_type = generic_type_mapping.get(Tuple, marshmallow.fields.Tuple)
            tuple_type = type_mapping.get(Tuple, marshmallow.fields.Tuple)
CedricCabessa

comment created time in 5 days

Pull request review commentlovasoa/marshmallow_dataclass

Allow to map a custom Field class to a Type

 def field_for_schema(     origin = typing_inspect.get_origin(typ)     if origin:         arguments = typing_inspect.get_args(typ, True)+        # Override base_schema.TYPE_MAPPING to change the class used for generic types below+        generic_type_mapping = base_schema.TYPE_MAPPING if base_schema else {}+         if origin in (list, List):             child_type = field_for_schema(arguments[0], base_schema=base_schema)-            return marshmallow.fields.List(child_type, **metadata)+            list_type = generic_type_mapping.get(List, marshmallow.fields.List)
            list_type = type_mapping.get(List, marshmallow.fields.List)
CedricCabessa

comment created time in 5 days

Pull request review commentlovasoa/marshmallow_dataclass

Allow to map a custom Field class to a Type

 def field_for_schema(     origin = typing_inspect.get_origin(typ)     if origin:         arguments = typing_inspect.get_args(typ, True)+        # Override base_schema.TYPE_MAPPING to change the class used for generic types below+        generic_type_mapping = base_schema.TYPE_MAPPING if base_schema else {}+
        # Override base_schema.TYPE_MAPPING to change the class used for generic types below
        type_mapping = base_schema.TYPE_MAPPING if base_schema else {}

CedricCabessa

comment created time in 5 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha 6468ebca5fcd5650db68e9a7a93452f4976428ba

Create releases automatically from master

view details

push time in 5 days

Pull request review commentlovasoa/marshmallow_dataclass

Allow to map a custom Field class to a Type

 def field_for_schema(     origin = typing_inspect.get_origin(typ)     if origin:         arguments = typing_inspect.get_args(typ, True)+        generic_type_mapping = {+            **DEFAULT_GENERIC_TYPE_MAPPING,+            **(base_schema.TYPE_MAPPING if base_schema else {}),+        }         if origin in (list, List):-            child_type = field_for_schema(arguments[0], base_schema=base_schema)-            return marshmallow.fields.List(child_type, **metadata)+            child_type = field_for_schema(+                arguments[0], base_schema=base_schema+            )+            list_type = generic_type_mapping[List]+            return list_type(child_type, **metadata)

I understand it's a tradeoff, but I think the code will be both easier to read and more performant with .get(key, default)

CedricCabessa

comment created time in 5 days

issue closedlovasoa/dezoomify-rs

Problem with DIGitalARhiiv

Good morning, @lovasoa: I hope You're fine and safe 🤗. I don't understand how create info.json or .yaml files for the maps on DIGAR Archiv. F.e., this one. Honestly, I don't know if this is something more inherent to dezoomify, dezoomify-rs or the dezoomify browser extension🤔: I opened the issue here 'cause the images in DIGAR are often very large format. Thank You in advance.

closed time in 5 days

Arekkusu1998

issue commentlovasoa/dezoomify-rs

Problem with DIGitalARhiiv

You can use the generic dezoomer, either in dezoomify or dezoomify-rs :

https://www.digar.ee/viewer/slice/16012/21518/9796/0/{{X}}/{{Y}}

dezoomified

Arekkusu1998

comment created time in 5 days

Pull request review commentlovasoa/marshmallow_dataclass

Allow to map a custom Field class to a Type

 def field_for_schema(     origin = typing_inspect.get_origin(typ)     if origin:         arguments = typing_inspect.get_args(typ, True)+        generic_type_mapping = {+            **DEFAULT_GENERIC_TYPE_MAPPING,+            **(base_schema.TYPE_MAPPING if base_schema else {}),+        }         if origin in (list, List):-            child_type = field_for_schema(arguments[0], base_schema=base_schema)-            return marshmallow.fields.List(child_type, **metadata)+            child_type = field_for_schema(+                arguments[0], base_schema=base_schema+            )+            list_type = generic_type_mapping[List]+            return list_type(child_type, **metadata)

Do we really need DEFAULT_GENERIC_TYPE_MAPPING ? Could we just use type_mapping.get(List, marshmallow.fields.List) ?

CedricCabessa

comment created time in 5 days

issue commentlovasoa/dezoomify-rs

Unable to parse DZI file with a Byte Order Mark

OK, I implemented a workaround for the underlying issue. I'll soon release a new version of dezoomify-rs, and you'll be able to use it with your file.

macmacleannz

comment created time in 5 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha 07068f7f7d2ecd690dad8b483b1441851e813be4

Add a workaround for https://github.com/netvl/xml-rs/issues/155 Fixes #45

view details

Ophir LOJKINE

commit sha 54b2b8e23a0455b0af002f20415d02108a1700b7

Save images to PNG be default when they don't fit JPEG JPEG can't store images wider or higher than 65535 pixels, so when such a file arrives, store it as PNG. This will create very large files, but at least won't fail after having downloaded all the tiles.

view details

push time in 5 days

issue closedlovasoa/dezoomify-rs

Unable to parse DZI file with a Byte Order Mark

I am getting the following error using dezoomify-rs

ERROR Dezoomer error: Unable to create the dezoomer: Tried all of the dezoomers, none succeeded. They returned the following errors:

 - deepzoom: Unable to create the dezoomer: Unable to parse the dzi file: Syntax: 1:1 Unexpected characters outside the root element: 

Using the same URL I have no problem using Dezoomify on the web

http://www.cartomundi.fr/site/cmfiles/Cartes/SP8/d_Eg-124.jpg/dzc_output.xml

Any clues on what I am doing wrong?

Thanks Mac

closed time in 5 days

macmacleannz

issue commentnetvl/xml-rs

Parsing XML preceded by Unicode BOM triggers an error

Maybe I misunderstood your comment on the original PR ? You were not proposing a wrapper that would be internal to xml-rs ? How would you then fix the bug when parsing XML from a str, which may also start with a BOM character ?

dylanmckay

comment created time in 5 days

issue commentnetvl/xml-rs

Parsing XML preceded by Unicode BOM triggers an error

it would require changing types in the API

Why is that exactly ? Why would we need to expose this wrapper ? It should be completely transparent. The read wrapper will be created and stored as a private field in EventReader. Is there something that I do not see ?

The problem with having an external crate is that most people who want to parse XML in rust are simply not aware of this bug. They are not actively looking for ways to work around it. Just like me, they write their program, test it, everything works, and some time later, in production, the code encounters an xml string that starts with a BOM, and fails. I really think that this library is the right place to fix it, and I'm ready to help.

dylanmckay

comment created time in 5 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha ad6c3b97bd8bb353d4de584b9ba4d83663fb9200

Prevent a single failed tile download to cause buffering of the whole image

view details

push time in 5 days

issue commentnetvl/xml-rs

Parsing XML preceded by Unicode BOM triggers an error

If you agree to merge this temporary solution, you can just close this bug today and hopefully never have to reopen it.

dylanmckay

comment created time in 6 days

issue commentnetvl/xml-rs

Parsing XML preceded by Unicode BOM triggers an error

Yes, I understand I can work around this bug in my own programs, but I was proposing to work on a patch that will fix it for everyone, since it seems to affect many users. Of course, once you merge your branch adding support for other encodings, you can just ditch this Read wrapper.

dylanmckay

comment created time in 6 days

issue commentlovasoa/dezoomify-rs

Unable to parse DZI file with a Byte Order Mark

I can add a workaround for this bug... In the meantime, there is still the node version of dezoomify

macmacleannz

comment created time in 6 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha eb68362f9ab22e30a3b30f513fe38cda403c3679

Update dependencies

view details

push time in 6 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha 126ceff008cd7efed7ead1e72f23f3a7c04cbf1f

Avoid unnecessary memory copies in the pixel streamer (used for PNGs) This should reduce memory fragmentation and slightly improve performance

view details

push time in 6 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha a778704f5ebefea8c3d21b5e558ba7b403a86416

Add expected png for tests

view details

Ophir LOJKINE

commit sha 425eefaa85ebba55a1e9be96ef7411cd5def17c6

Write PNGs in larger chunks for better performance

view details

push time in 6 days

push eventlovasoa/dezoomify-rs

Ophir LOJKINE

commit sha 9966051ee79c7482431968bb7547299449b9aa40

Add benchmarks to evaluate the performance changes

view details

Ophir LOJKINE

commit sha 06424805d4459646d04def3e73408b0585799fb1

Merge remote-tracking branch 'origin/master'

view details

push time in 6 days

more