profile
viewpoint
Dominic Jack jackd Queensland University of Technology Brisbane jackd.github.io PhD Candidate in Deep Learning / Computer Vision

jackd/human_pose_util 23

Utility files for human pose estimation in python

jackd/adversarially_parameterized_optimization 14

GAN-based 3D human pose estimation model for 3DV'17 paper

jackd/ifp-sample 4

Cython iterative farthest point sampling implementation

jackd/dids 2

Dictionary Interface to DataSets for python

jackd/antlr4dart-runtime 1

ANTLR4 runtime library for Dart.

jackd/crfrnn 1

Tensorflow implementation of CrfRnn layer

jackd/datasets 1

A collection of datasets ready to use with TensorFlow

jackd/events-tfds 1

tensorflow_datasets implementations for events datasets

jackd/ffd-tf2 1

template deformation network using tensorflow 2.0 and keras

jackd/FVMPor 1

Finite volume method for simulating fluid flow and associated processes in porous media

pull request commenttensorflow/datasets

[updated PR] added mpii human pose dataset

@googlebot I consent

hermannsblum

comment created time in 15 days

pull request commenttensorflow/datasets

[updated PR] added mpii human pose dataset

@hermannsblum go for it, I never worked out what tfds needed to get this through and I'm working on very different stuff now.

hermannsblum

comment created time in 15 days

issue openedtensorflow/tensorflow

keras.Model.fit cannot handle variable epoch sizes

<em>Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template</em>

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): ubuntu 18.04
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): v2.2.0-0-g2b96f3662b 2.2.0
  • Python version: 3.6.9

Describe the current behavior keras.Model.fit with finite-length dataset inputs can run without steps_per_epoch. However, if the number of batches in the epoch changes between epochs, this can lead to:

  • truncation of epoch if subsequence epochs are longer than the initial epoch; and
  • truncation of training if subsequence epochsa re shorter than the initial epoch.

While not a common occurance, this happens e.g. in graph neural network applications where it makes more sense to batch according to some maximum number of edges/nodes in a graph as opposed to a fixed batch size. A fixed number of graphs in a dataset can result in a variable number of batches when these are shuffled.

Describe the expected behavior Each epoch of fit should iterate over a dataset in it's entirety when steps_per_epoch is not provided, and run to the defined number of epochs (unless terminated prior with callbacks), even if the number of steps varies between epochs. Alternatively, a special steps_per_epoch value (say, -1) should be able to be provided to specify a dynamic epoch size. Standalone code to reproduce the issue Colab

Other info / logs From colab above, when subsequence epochs are shorter than the first:

WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 30 batches). You may need to use the repeat() function when building your dataset.

created time in a month

push eventjackd/kblocks

Dominic Jack

commit sha ee71f6a7ec97b4a084149e79ba20ab151d144eed

added license

view details

push time in 2 months

issue commentjackd/more-keras

LICENSE

Hi @Evizero, my latest iteration of this is kblocks. I'll throw an MIT license up over there now...

Evizero

comment created time in 2 months

push eventjackd/tensorflow

Dominic Jack

commit sha 57eccc7bc29ddb105dcaa2f6a413163461ad9987

added test

view details

push time in 2 months

push eventjackd/numba-neighbors

Huite

commit sha 9145e057384951f5869eb7b99cfbbc644bbf0311

Add explicit license

view details

Dominic Jack

commit sha c8eca4f177e7e39b8dfc77869b0f3b51ab91121c

Merge pull request #1 from Huite/master Add explicit MIT license to repo

view details

push time in 2 months

PR merged jackd/numba-neighbors

Add explicit MIT license to repo

Very minor PR, mostly an excuse to get in touch!

I was looking at some KDTree implementations, especially one that works well with numba. I found your comment in this issue: https://github.com/numba/numba-scipy/issues/36 ... which obviously brought me here!

Some background: I'm coming from a geospatial use case, I want to do some (2D for now) unstructured mesh regridding which involves computing some areal overlaps from overlapping cells. Most (vector-based) GIS stuff seems awfully slow or some existing solutions come with massive binary dependencies, so I figured I'd have a look at it myself. A KDTree with a radius query seems like a decent starting point to create some sort of short list of possibly overlapping cells. I greatly prefer numba since it's basically seamless to Python, and it's much nicer to distribute than e.g. Cython; the JIT'ing also allows arbritary area weighting functions at runtime, so it's great all around.

Anyway, numba-neighbors is looking pretty spiffy! I've been looking at these:

  • https://github.com/storpipfugl/pykdtree
  • https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.cKDTree.html
  • https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KDTree.html

And for my ad hoc benchmark (but fairly realistic for the use case), numba-neighbors beats them by a fairly wide margin (> 30% on cKTree, more so on the others). Might be interesting to redo this analysis at some point: https://jakevdp.github.io/blog/2013/04/29/benchmarking-nearest-neighbor-searches-in-python/

Anyway, my main issue with the other methods is how they return the indices. I don't really want to be stuck with a fixed number of neighbors, so sklearn's radius_query is what I want... except it returns an array of arrays which probably isn't going to be great for further numba functions to work on. I could copy and edit some Cython stuff, but as mentioned numba is just much nicer to distribute. So with numba-neighbors it should be pretty easy to write a custom query function for my goals!

Wrapping up: numba-neigbors seems useful, I'd like to use it. I saw you did put license='MIT' in the setup.py -- but I had to look for it (if only briefly).

+21 -0

1 comment

1 changed file

Huite

pr closed time in 2 months

pull request commentjackd/numba-neighbors

Add explicit MIT license to repo

Thanks for getting in touch / fixing this up - glad to hear you're finding it useful. This was very much a "I have code I developed for another project, might as well open source it" rather than "let's build a general-purpose KDTree library", and I opened it up in a bit of a rush. There'll no doubt be other things like this, and happy to hear thoughts on the interface. I'm a little tied up with other things at the moment to give this a thorough cleaning, but if you notice anything else dodgy/confusing I'll keep an eye on PRs :).

So with numba-neighbors it should be pretty easy to write a custom query function for my goals!

Note the BinaryTree class has a query_radius method. It's fairly similar to those other interfaces, except (a) you need to specify a max_count for pre-allocation purposes (and if you under-estimate the returned indices are not necessarily the closest); and (b) it doesn't sort the returned indices by distance.

I originally wrote wrappers that solved each of these issues, but they slowed things down and weren't necessary for the use case I developed it for.

Huite

comment created time in 2 months

issue commentjackd/pointnet_ops

fail to make pip_pkg

Do you have write access to pointnet_ops directory?

boboofan

comment created time in 2 months

PR opened tensorflow/tensorflow

ensure model initialized on ANY trackable attr set

In particular, empty tuples should not trigger this.

+1 -1

0 comment

1 changed file

pr created time in 2 months

create barnchjackd/tensorflow

branch : keras-model-attrs

created branch time in 2 months

push eventjackd/tensorflow

A. Unique TensorFlower

commit sha 82143c1ad88ceeb51ac6a280b79c2bc766dd854b

Enable TensorCore embeddings for training via FeatureColumnV2. PiperOrigin-RevId: 312340625 Change-Id: I559aba797a8f1a37ecec1e4ee71cd027701ae6dd

view details

Ajay P

commit sha deaf5ea06a618749113f7def6285ebf77c19dfa7

Addressed PR comments

view details

Tomer Kaftan

commit sha 935c55c590898f589de230c60ccbc6d50f09a8c7

Fix performance regression involving trainable check in batchnorm. The regression made it so a tf `and` op would be used for something that just required a python check, which in turn would make an if statement build a tf.cond instead of a python if. This change makes it just use a python if. PiperOrigin-RevId: 312345759 Change-Id: I568c9c992287bfc3e693f34b7b51bd7f35388f34

view details

Tamas Bela Feher

commit sha dc0149adb93c612f2c1f7fb6fc294ebcfee8e36a

Test ConvertActivation in dynamic shape mode Additionally helper functions for dynamic shape tests are improved: - Added test parameter member variables to ParameterizedOpConverterTestBase - Single TetOpConverter function introduced - Type parameter handling for input tensors simplified

view details

Mihai Maruseac

commit sha baa3e80ca55909ce1b56864c0adcd825862fea63

Fix pylint errors PiperOrigin-RevId: 312356181 Change-Id: I7259f4e5584e2947dcf919918a5e9371e50185fe

view details

Dan Moldovan

commit sha 53215ab702aedb306778590277f71768c2d2c148

Use the nonlocal mechanism for if statements. This is the same mechanism used by for and while loops and it allows reusing much of the code. This required the ternary if operator to be split in a separate implementation, but that better accounts for its different nature. This should also allow more consistent verification and error messages throughout. PiperOrigin-RevId: 312360755 Change-Id: I57989c6cd40a16653521e18ccf21f2b0e994bd96

view details

Mihai Maruseac

commit sha af2263101b0407ff1fb7f7e492565f1edc4cee30

Don't try t gather from empty tensors PiperOrigin-RevId: 312361331 Change-Id: I8f81add090d9a5452e671c48a03e0f5cb9f81a41

view details

Lucy Fox

commit sha ca53894d61ca46e3d6a007a6de0c8c3458ead931

Canonicalize tf.Select to tf.SelectV2. The ops are mostly equivalent, except that Select has stricter requirements and does not support broadcasting, whereas SelectV2 does. There is one special case to be considered in this canonicalization, which is when the predicate is a tensor and the data is multidimensional. In this case, Select op semantics dictate that the predicate tensor length must match the size of the first data dimension. This varies from normal broadcasting semantics, which are used in SelectV2, so we must reshape the tensor in this case to be compatible. This also adds verifiers and tests for the Select and SelectV2 ops in the MLIR TF dialect. PiperOrigin-RevId: 312362580 Change-Id: I43f326ad330c92ce279b25cecf5a2cf46714ce3f

view details

Karim Nosir

commit sha 930709e46e3b80345ffeac92b7728873dc97b0a7

Hexagon delegate: support SLICE op for uint8/int8 PiperOrigin-RevId: 312362625 Change-Id: Idf0185c33bc7d64cf480a70166c15f64b0a409b1

view details

A. Unique TensorFlower

commit sha d894109fe1203f2259819841b85a0354c7780609

Fix parameter check for batchdim in Gather. PiperOrigin-RevId: 312371119 Change-Id: I7537194147199136b5b847ce6d1ddd361e42a393

view details

A. Unique TensorFlower

commit sha 91da977a0305f2c25c6c149a5924d8b1eb33375c

Improve the bucket generation in Boosted Trees to avoid returning more than requested buckets. PiperOrigin-RevId: 312371738 Change-Id: I7f241c839f52d679ad4ceb82c161018e9b944fa3

view details

Sachin Joglekar

commit sha 119aa03c7605a82061227c3291e8c5665752c90f

Split delegate-specific interpreter tests into a separate file PiperOrigin-RevId: 312372505 Change-Id: If366a884ce090f2ad40bdc20d266ef32eb5a1765

view details

A. Unique TensorFlower

commit sha f0eb6dff6fb4f0500f45c8ca2b82c365de17f403

adjust gather ops launch config. for NCF model, this means ~20% gain. (due to grid size from 80->160 on volta). PiperOrigin-RevId: 312373706 Change-Id: I2413d301ec170e6e90eeae025e4bb17fccd5abbb

view details

A. Unique TensorFlower

commit sha db573482f4f8712ff173f9bb511dceef4128228c

Add a folder for numpy API PiperOrigin-RevId: 312374580 Change-Id: Ic8be81b738659668814e956d7fd4da7972944257

view details

Ajay P

commit sha e5fcb88fa425f6c905d5bbb28d0da2bfa6257587

Fixed sanity test issue

view details

A. Unique TensorFlower

commit sha f8a918ccf6d39aa6c1dbf56716f1bd476322c1cd

Allow tile op to work on variant dtype. PiperOrigin-RevId: 312382133 Change-Id: I3a0f95865ca0f782fa73f7ba55b3d987de006332

view details

A. Unique TensorFlower

commit sha c9819edcfc30424c52e24c4288ccf9fd7df30b63

pfor: Enable handling VariableShape PiperOrigin-RevId: 312382248 Change-Id: Ie8f600dadeb6f5ef3bb7483f7435348ac002e176

view details

Jiho Choi

commit sha 6fad0820f83b8d90fb33acad4589563b479dbd73

Use enum instead of string to specify the context type. PiperOrigin-RevId: 312385361 Change-Id: I7ee9c203e7f662bf6898d80d9fe2b75536d0e044

view details

Hye Soo Yang

commit sha 692bb1da53493a6cf37dc28a4c1e1a82df32d9fa

Fix nightly breakage (macos, linux). PiperOrigin-RevId: 312388059 Change-Id: I9db7d5e73b82298df3f800ece69cdccd92e706ca

view details

A. Unique TensorFlower

commit sha 7f3ef3e1eae4d7142cb3b52ede78caa18a37e96c

[Profiler] Add optimization advice for reducing the host-to-TPU data transfer time. PiperOrigin-RevId: 312388184 Change-Id: I2fc8a60af6724467e447026dde7a8d6925ed1357

view details

push time in 2 months

pull request commenttensorflow/community

RFC: CSRSparseMatrix

@penpornk is there a reason this isn't exposed as a CompositeTensor?

Separately, can I request the following ops are considered/exposed in python?

  • constructor that takes components (values, column_indices, row_pointers, batch_pointers)
  • properties that return components
  • CSRMatrix.with_values(new_values) that creates a new matrix with the same sparsity but with values replaced with new_values, similar to RaggedTensor.with_values
penpornk

comment created time in 2 months

pull request commenttensorflow/community

RFC: CSRSparseMatrix

@Padarn tf-graphics has some use-cases with graph neural networks, though they're currently implemented with tf.gather + tf.reduce_sum . Performance benefits using COO are inconsistent. See this PR for equivalent implementations and the discussion for rough benchmarks.

penpornk

comment created time in 2 months

pull request commenttensorflow/community

RFC: CSRSparseMatrix

Would there be benefit in implementing indices as a RaggedTensor, rather than col_indices, row_pointers, batch_pointers? I'm not sure if RaggedTensors are a thing in the c++ API, but I imagine this would make a lot of things easier in the python API, e.g. batching?

penpornk

comment created time in 3 months

issue openedcgtuebingen/Flex-Convolution

Supplementary material

I found the supplementary material for the paper a while back, but can't seem to find it any more. Is it still available somewhere?

created time in 3 months

PR closed tensorflow/graphics

optimized/debugged data pipeline cla: yes

Small adjustments to pointnet data pipeline.

A couple of things that will make a difference to performance under certain configurations:

  • fixed bug with point cropping (original cropping was done on the batch dimension)
  • removed data augmentation from test set, even when data augmentation is turned on (as per original implementation).

Optimizations:

  • moved preprocessing ops into tf.data.Dataset.map
  • added prefetch

I don't imagine these optimizations will provide an observable difference in this case, but I think it is appropriate nonetheless if this is to be used as an example.

+26 -13

5 comments

1 changed file

jackd

pr closed time in 3 months

pull request commenttensorflow/graphics

keras Model.fit pointnet v1

I like to have a clear separation between data stored on disk (from tfds) and preprocessing with tf.data, e.g. I don't think it's appropriate for jittering to be a part of the builder/config itself, but perhaps a preprocessing file alongside it. I guess it depends on whether or not you feel preprocessing is a property of the data or part of the model. I tend to feel its part of the model (though even then it's not clear cut. e.g. I feel the number of points is a property of the the data, while the jittering is a property of the model).

jackd

comment created time in 3 months

pull request commenttensorflow/graphics

optimized/debugged data pipeline

@ataiya if #312 isn't too big to do in one PR I might abandon this one - I've already resolved merge conflicts there, will address your comments there shortly. Happy to do both PRs if necessary, just don't want to waste your time.

jackd

comment created time in 3 months

pull request commenttensorflow/graphics

keras Model.fit pointnet v1

@ataiya I'm happy to duplicate learning rate / preprocessing across examples. The only reason I'm tempted to keep at least data preprocessing separate is for reusability, e.g. it's nice to be able to say with confidence "we used exactly the same data preprocessing as pointnet" without having to copy/paste from examples. I agree helpers.py wouldn't be the appropriate place for that, but perhaps a small preprocessing.py file?

jackd

comment created time in 3 months

Pull request review commenttensorflow/graphics

keras Model.fit pointnet v1

   __all__.remove("version") # pylint: enable=g-statement-before-imports,g-import-not-at-top -__version__ = "HEAD"

I was wondering where I accidentally managed to change that...

jackd

comment created time in 3 months

push eventjackd/graphics

Dominic Jack

commit sha cccb4f320a0e4aa40a10ea0135e43bab34a55579

undid version change

view details

push time in 3 months

issue commenttensorflow/graphics

Use TFDS mock_data for pointnet_test

@ataiya PR is merged, but I can't close.

ataiya

comment created time in 3 months

push eventjackd/graphics

A. Unique TensorFlower

commit sha ed7173eccc5cbb6296da27557f9ca782f52f20e8

Updates TF Graphics metadata PiperOrigin-RevId: 304701489

view details

Dominic Jack

commit sha df90fed7eb95b540783286167c3b9e0fc60be371

added metadata dir for tfds.mock_data

view details

Dominic Jack

commit sha edab69c2101bf1899255c708ac6dfb5093fa8da0

relaxed logging to empty directory

view details

Dominic Jack

commit sha 7b81d466570f094a486abb4b7c29d5b1cac54a5e

Added training/evaluation testing with mock_data

view details

Dominic Jack

commit sha 837e742834f91a739838c44274c9f80626c75c83

fixed sys.argv

view details

Dominic Jack

commit sha 28816e1f37ea2d552f873f00b07ead4cc26dfd41

moved mock_data to test

view details

Dominic Jack

commit sha 078cac65c647e48c061555adf8d27046e86d1837

fix evaluation at end, not per step

view details

Dominic Jack

commit sha 1a36e902d578d787d0674c92caa1b042ae40bf45

addressing comments

view details

Dominic Jack

commit sha c2f100875432cd2609eb7c0d89e80aae0f103539

reformat, remove unused tfds flag

view details

Dominic Jack

commit sha 55d48eac2a35092281bf85636151e8685869fa6e

formatting clargs

view details

Andrea Tagliasacchi

commit sha dbf8fb761652a0f7e062a0b2320624724f085e50

Deployed github workflow (path fix) PiperOrigin-RevId: 310838670

view details

Dmitry Volodin

commit sha 8f17880b49a0a5e458b825b843e20e4322889d01

typo

view details

Aviv Segall

commit sha 84195a0c6146f94a3f7eae395230e61121ed8eb4

Add Shapenet (ShapeNetCore.v2) dataset to tensorflow_datasets. PiperOrigin-RevId: 310935648

view details

Andrea Tagliasacchi

commit sha af7f2f9ddeeaa81e18cbf7f86b2778729bc3e17b

Fix to workflow PiperOrigin-RevId: 310980875

view details

Cem Keskin

commit sha 5d902d033e9f0e3c8fce820eaa95f180881d3bc9

Exposes eps argument for tfg.math.cartesian_to_spherical_coordinates. PiperOrigin-RevId: 311384397

view details

Julien Valentin

commit sha 0236a7f95d4f15d93da1081f7aafeb5609440dd7

Fix the travis widget to reflect the status of master. PiperOrigin-RevId: 311482619

view details

Copybara-Service

commit sha 6800317e8095a07ffb915f482a122959e2adb3f3

Merge pull request #222 from tensorflow:test_304701489 PiperOrigin-RevId: 311745638

view details

Aviv Segall

commit sha 5d35e72a8844b99192510402d2e3a7c7157663fb

Refactor the Shapenet dataset to support different BUILDER_CONFIGS. PiperOrigin-RevId: 311760835

view details

Andrea Tagliasacchi

commit sha 2a759c0981372cc58e51e77ba939b00693cb9fdf

Updated travis to avoid internal force-push

view details

Andrea Tagliasacchi

commit sha 87b673582aee03c63d32e6667fc76b1ff1dadaa4

Update to auto-documentation workflow. PiperOrigin-RevId: 311987485

view details

push time in 3 months

delete branch jackd/graphics

delete branch : pointnet-mock-data-test

delete time in 3 months

pull request commenttensorflow/graphics

Pointnet mock data test

@julienvalentin is that the final review stage? Am I right to merge (or squash and merge)? I don't want to break anything...

jackd

comment created time in 3 months

issue commenttensorflow/graphics

Roadmap – feedback welcome!

Meeting recording is private :(

osanseviero

comment created time in 3 months

pull request commenttensorflow/graphics

keras Model.fit pointnet v1

@julienvalentin once #310 is through I'll ensure tests are passing, then should be good for review.

@ataiya should I make a separate utils.py file for shared dataset/learning rate functionality shared between fit and train? They're currently in helpers.py, but they're a little project specific...

jackd

comment created time in 3 months

pull request commenttensorflow/graphics

Pointnet mock data test

@ataiya am I right to go ahead and merge this now? The button has gone green, but I don't want to break anything right off the bat... :)

jackd

comment created time in 3 months

Pull request review commenttensorflow/graphics

optimized/debugged data pipeline

 # limitations under the License. """Training loop for PointNet v1 on modelnet40.""" # pylint: disable=missing-function-docstring-+import functools import tensorflow as tf +from tqdm import tqdm from tensorflow_graphics.datasets.modelnet40 import ModelNet40 from tensorflow_graphics.nn.layer.pointnet import PointNetVanillaClassifier as PointNet-from tensorflow_graphics.projects.pointnet import augment+from tensorflow_graphics.projects.pointnet import augment as augment_lib

rename / split done. Implementation does NOT rotate the test data.

jackd

comment created time in 3 months

pull request commenttensorflow/graphics

optimized/debugged data pipeline

@julienvalentin sorry, got distracted and never got around to addressing comments. I've just made a small tweak as requested by @ataiya (augment -> (augment_rotation, augment_jitter)). I see I can now merge - fairly sure that wasn't an option before. Is that because I accepted contributer invite, or has this been reviewed?

If it's the former and there's still a review process to be had, #312 is a larger PR that encompasses these changes, among other things. The tests there are currently failing just because I'm waiting on #310 . atayia has already gone over much of that. If the overhead to starting a review is high I'd suggest starting there and dropping this one, but if you'd rather smaller PRs then this one's ready to go, and I'll manage the merge conflicts with 312 later.

jackd

comment created time in 3 months

push eventjackd/graphics

Dominic Jack

commit sha b438f322a84d7d2bda4da077ef93a8f1691093d1

tweaked augmentation / preprocessing

view details

push time in 3 months

more