profile
viewpoint

Thomas-K-John/SalesForecasting 1

Sales Forecasting of Retail Clothing Product Categories

INSOFE-B30/decision-tree-Thomas-K-John 0

decision-tree-Thomas-K-John created by GitHub Classroom

Thomas-K-John/Detection-and-Prediction-on-Phishing-Websites 0

The aim is to predict the type of the website in an automated manner to either "legitimate" or "phishy" class labels

issue openedtensorflow/tensorflow

Handling Unbalanced dataset : ImageAugmentation using tf.keras.preprocessing.image.ImageDataGenerator and tf.datasets: model.fit()

URL(s) with the issue:

https://www.tensorflow.org/guide/keras/train_and_evaluate#using_sample_weighting_and_class_weighting

https://www.tensorflow.org/tutorials/images/classification#data_augmentation

Description of the issue (what needs changing):

According to the TensorFlow documentation we are not supposed to use class_weights parameter when training the models (model.fit()) using tf.datasets. So the suggested way is to use sample_weights. I was able to use sample_weights and create a dataset using sample_weights and passed it to the model.fit() method without using ImageDatGenerators.

a) But how can we do the same while using ImageDataGenerator class and flow_from_dataframe() data generator and using tf.datasets?

b) Is there any option to pass sample_weights in tf.keras.preprocessing.image.ImageDataGenerator class's flow()/flow_from_dataframe() methods?

c) The second URL which I have shared above also does not mention whether the ImageDataGenerator class has a way of balancing the unbalanced datasets.

Usage example

Can you provide a usage example?

created time in 2 months

issue commenttensorflow/tensorflow

ImageAugmentation using tf.keras.preprocessing.image.ImageDataGenerator and tf.datasets: model.fit() is running infinitely

I hope this fix will also resolve the issue faced with predict() and evaluate() methods.

Thomas-K-John

comment created time in 2 months

push eventThomas-K-John/MLApp

Thomasaws1

commit sha bd2a03a7b6cecebb558565ebc26350bb8024b168

Update requirements.txt

view details

Thomasaws1

commit sha e0999a8b73ee723effe5b8d59b82523de83ce2e0

T3

view details

Thomas K John

commit sha 76a86f25f42e14a1f4d1da35ee5dba3ef5ed3da8

Merge pull request #1 from Thomasaws1/master Test

view details

push time in 2 months

PR merged Thomas-K-John/MLApp

Test
+2 -2

0 comment

1 changed file

Thomasaws1

pr closed time in 2 months

push eventThomas-K-John/MLApp

Thomas K John

commit sha 72c26c1ec9daf309c6d9ff12d450c9b1050f1cf8

First commit

view details

push time in 2 months

create barnchThomas-K-John/MLApp

branch : master

created branch time in 2 months

created repositoryThomas-K-John/MLApp

created time in 2 months

issue commenttensorflow/tensorflow

ImageAugmentation using tf.keras.preprocessing.image.ImageDataGenerator and tf.datasets: model.fit() is running infinitely

Few points that I found from my testing:

  1. As you suggested I tried by leaving the batch size unspecified (as None) in from_generator. This helped me to successfully train the model by successfully running the model.fit() method. But I was not able to use this model for making any predictions. i.e. model.predict() and model.evaluate() methods were just running infinitely again.

  2. I also tried to pass the validation dataset to my fit() method during the training of the model. The model was again running infinitely. Please find the arguments passed to my fit() method below: model.fit(train_ds, verbose=1, steps_per_epoch=3670/32, validation_data=validation_ds, epochs =1)

I would like you to please confirm if these two are bugs. Please let me know if I need to raise any seperate bugs of these issues. I would like you to please suggest me any work around to run the model.predict() and model.evaluate() functions.

Thomas-K-John

comment created time in 2 months

issue commenttensorflow/tensorflow

ImageAugmentation using tf.keras.preprocessing.image.ImageDataGenerator and tf.datasets: model.fit() is running infinitely

I tried to use the model.fit() method by passing the steps_per_epoch=3670/32 and I got the following error: Code used to run the fit() method used: model.fit(ds, verbose=1, steps_per_epoch=3670/32, epochs =1) Error: 112/114 [============================>.] - ETA: 25s - loss: 1.9140 - accuracy: 0.3262

InvalidArgumentError Traceback (most recent call last) <ipython-input-13-e56f4562ba98> in <module>() ----> 1 model.fit(ds, verbose=1, steps_per_epoch=3670/32, epochs =1)

8 frames /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name) 58 ctx.ensure_initialized() 59 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, ---> 60 inputs, attrs, num_outputs) 61 except core._NotOkStatusException as e: 62 if name is not None:

InvalidArgumentError: ValueError: generator yielded an element of shape (22, 256, 256, 3) where an element of shape (32, 256, 256, 3) was expected. Traceback (most recent call last):

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 243, in call ret = func(*args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper return func(*args, **kwargs)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 821, in generator_py_func "of shape %s was expected." % (ret_array.shape, expected_shape))

ValueError: generator yielded an element of shape (22, 256, 256, 3) where an element of shape (32, 256, 256, 3) was expected.

 [[{{node PyFunc}}]]
 [[IteratorGetNext]] [Op:__inference_train_function_1244]

Function call stack: train_function

NOTE: I do understand that when we have 3670 training records, the last step(114th) will contain only 22 records and that should not throw an errors. Please correct me if I am wrong.

Thomas-K-John

comment created time in 2 months

issue openedtensorflow/tensorflow

ImageAugmentation using tf.keras.preprocessing.image.ImageDataGenerator and tf.datasets: model.fit() is running infinitely

What I need help with / What I was wondering I am facing issue while running the fit() function in TensorFlow(v 2.2.0-rc4) with augmented images(using ImageDataGenerator) passed as a dataset. The fit() function is running infinitely without stopping.

What I've tried so far I tried it with the default code which was shared in Tensorflow documentation.

Please find the code snippet below:

import tensorflow as tf from tensorflow.keras.preprocessing.image import ImageDataGenerator from tensorflow.keras.models import Sequential, Model from tensorflow.keras.layers import Dense, Dropout, Flatten from tensorflow.keras.layers import Conv2D, MaxPooling2D from tensorflow.keras.layers import Input, Dense

flowers = tf.keras.utils.get_file( 'flower_photos', 'https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz', untar=True)

img_gen = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255, rotation_range=20)

images, labels = next(img_gen.flow_from_directory(flowers))

print(images.dtype, images.shape) print(labels.dtype, labels.shape)

train_data_gen = img_gen.flow_from_directory( batch_size=32, directory=flowers, shuffle=True, target_size=(256, 256), class_mode='categorical')

ds = tf.data.Dataset.from_generator(lambda: train_data_gen, output_types=(tf.float32, tf.float32), output_shapes=([32, 256, 256, 3], [32, 5]) )

ds = ds.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)

it = iter(ds) batch = next(it) print(batch)

def create_model(): model = Sequential() model.add(Conv2D(32, (3, 3), activation='relu', input_shape=images[0].shape)) model.add(Conv2D(32, (3, 3), activation='relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.5)) model.add(Conv2D(64, (3, 3), activation='relu')) model.add(Conv2D(64, (3, 3), activation='relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.5)) model.add(Flatten()) model.add(Dense(64, activation='relu')) model.add(Dropout(0.5)) model.add(Dense(5, activation='softmax')) return model

model = create_model() model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=["accuracy"]) model.fit(ds, verbose=1, batch_size= 32, epochs =1)

This last line of code - fit() is running infinitly without stopping. I had also tried passing steps_per_epoch = total_no_of_train_records/batch_size.

It would be nice if... I would like you to confirm whethere this is a bug in the tensorflow datasets package and in which release will this be fixed.

Environment information

  • System: Google colaborator
  • Python version: v3.6.9
  • tensorflow-datasets/tfds-nightly version: v2.2.0-rc4

created time in 2 months

more