profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/johngore123/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

johngore123/CLIP 0

Contrastive Language-Image Pretraining

johngore123/CogView 0

Text-to-Image generation

johngore123/DeepSpeed 0

DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.

issue commentlucidrains/DALLE-pytorch

horovod and OpenAI's Pretrained VAE are incompatible.

Does horovod work with taming for you?

wcshin-git

comment created time in a month

issue commentlucidrains/DALLE-pytorch

horovod and OpenAI's Pretrained VAE are incompatible.

Oh, really? I'm using RTX 3090 and when I try to use horovod with OpenAidVAE, I get an error like that...

Horovod seems to works best using multi-gpus, but it should still work on 1 so that might be an version error or something. error seems to hint at mpi begin the problem. so you may have an incompatible version.

wcshin-git

comment created time in a month

issue commentlucidrains/DALLE-pytorch

horovod and OpenAI's Pretrained VAE are incompatible.

I tried using 4 tesla v100's and it worked even with OpenAI's Pretrained VAE

wcshin-git

comment created time in a month

push eventjohngore123/DeepSpeed

johngore123

commit sha 5c38e120120381e68fb8cbc285c6be237e81922d

Update setup.py

view details

push time in a month

push eventjohngore123/DeepSpeed

johngore123

commit sha 6e5c86611cddd180d03d5e4341591dd7c03f5b1d

Update setup.py

view details

push time in a month

push eventjohngore123/DeepSpeed

johngore123

commit sha ac02d377ad9c9b830686b5f59ec07e1f3bac4056

Update install.sh

view details

push time in a month

push eventjohngore123/DeepSpeed

johngore123

commit sha 46220e098a8084d459cb930319379677b92a2885

Update install.sh

view details

push time in a month

push eventjohngore123/DeepSpeed

johngore123

commit sha 51c531cce1c98ad016a4bceb41745a745cfc0b10

Update install.sh

view details

push time in a month

push eventjohngore123/DeepSpeed

johngore123

commit sha 36d0872caecab99f4e93a072872bb8f982559c03

Update install.sh

view details

push time in a month

push eventjohngore123/DeepSpeed

johngore123

commit sha 781ecd4aa3b12fd23d6f1e410a8fd9ca6a61ec2a

Update install.sh

view details

push time in a month

fork johngore123/DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.

https://www.deepspeed.ai/

fork in a month

issue closedlucidrains/DALLE-pytorch

Got error '' Input tensor shape: torch.Size([1024]). Additional info: {'b': 4}. Expected 2 dimensions, got 1'' everything used to work fine now it keeps giving me this error, it only happens on taming.

deepspeed train_dalle.py --local_rank=0 --image_text_folder /home/valterjordan/DALLE-pytorch/datasets/train2017 --truncate_captions --deepspeed --distributed_backend deepspeed --fp16 --taming [2021-06-29 20:20:36,222] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only. [2021-06-29 20:20:36,232] [INFO] [runner.py:358:main] cmd = /home/valterjordan/miniconda3/envs/DALLE-pytorch/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=29500 train_dalle.py --local_rank=0 --image_text_folder /home/valterjordan/DALLE-pytorch/datasets/train2017 --truncate_captions --deepspeed --distributed_backend deepspeed --fp16 --taming

[2021-06-29 20:20:36,528] [INFO] [launch.py:80:main] WORLD INFO DICT: {'localhost': [0]} [2021-06-29 20:20:36,529] [INFO] [launch.py:89:main] nnodes=1, num_local_procs=1, node_rank=0 [2021-06-29 20:20:36,529] [INFO] [launch.py:101:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]}) [2021-06-29 20:20:36,529] [INFO] [launch.py:102:main] dist_world_size=1 [2021-06-29 20:20:36,529] [INFO] [launch.py:105:main] Setting CUDA_VISIBLE_DEVICES=0 /home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/pytorch_lightning/metrics/init.py:44: LightningDeprecationWarning: pytorch_lightning.metrics.* module has been renamed to torchmetrics.* and split off to its own package (https://github.com/PyTorchLightning/metrics) since v1.3 and will be removed in v1.5 "pytorch_lightning.metrics.* module has been renamed to torchmetrics.* and split off to its own package" Using DeepSpeed for distributed execution [2021-06-29 20:20:37,314] [INFO] [distributed.py:47:init_distributed] Initializing torch distributed with backend: nccl using pretrained VAE for encoding images to tokens Working with z of shape (1, 256, 16, 16) = 65536 dimensions. loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth VQLPIPSWithDiscriminator running with hinge loss. Loaded VQGAN from /home/valterjordan/.cache/dalle/vqgan.1024.model.ckpt and /home/valterjordan/.cache/dalle/vqgan.1024.config.yml 118287 image-text pairs found for training wandb: W&B syncing is set to offline in this directory. Run wandb online or set WANDB_MODE=online to enable cloud syncing. [2021-06-29 20:20:43,289] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.13+3352086, git-hash=3352086, git-branch=sparse_triton_support [2021-06-29 20:20:44,498] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 1, parameter_parallel_size: 1 [2021-06-29 20:20:44,513] [INFO] [engine.py:598:_configure_optimizer] Removing param_group that has no 'params'in the client Optimizer [2021-06-29 20:20:44,513] [INFO] [engine.py:602:_configure_optimizer] Using client Optimizer as basic optimizer [2021-06-29 20:20:44,513] [INFO] [engine.py:612:_configure_optimizer] DeepSpeed Basic Optimizer = Adam [2021-06-29 20:20:44,513] [INFO] [logging.py:60:log_dist] [Rank 0] Creating fp16 unfused optimizer with dynamic loss scale [2021-06-29 20:20:44,513] [INFO] [unfused_optimizer.py:36:init] Fused Lamb Legacy : False [2021-06-29 20:20:44,522] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed Final Optimizer = Adam [2021-06-29 20:20:44,522] [INFO] [engine.py:449:_configure_lr_scheduler] DeepSpeed using client LR scheduler [2021-06-29 20:20:44,522] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed LR Scheduler = None [2021-06-29 20:20:44,522] [INFO] [logging.py:60:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0003], mom=[(0.9, 0.999)] [2021-06-29 20:20:44,522] [INFO] [config.py:737:print] DeepSpeedEngine configuration: [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] activation_checkpointing_config { "contiguous_memory_optimization": false, "cpu_checkpointing": false, "number_checkpoints": null, "partition_activations": false, "profile": false, "synchronize_checkpoint_boundary": false } [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] allreduce_always_fp32 ........ False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] amp_enabled .................. False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] amp_params ................... {'opt_level': 'O1'} [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] checkpoint_tag_validation_enabled True [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] checkpoint_tag_validation_fail False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] disable_allgather ............ False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] dump_state ................... False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] dynamic_loss_scale_args ...... None [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] elasticity_enabled ........... False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] flops_profiler_config ........ { "detailed": true, "enabled": false, "module_depth": -1, "profile_step": 200, "top_modules": 1 } [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] fp16_enabled ................. True [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] global_rank .................. 0 [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] gradient_accumulation_steps .. 1 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] gradient_clipping ............ 0.5 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] gradient_predivide_factor .... 1.0 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] initial_dynamic_scale ........ 4294967296 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] loss_scale ................... 0 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] memory_breakdown ............. False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] optimizer_legacy_fusion ...... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] optimizer_name ............... None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] optimizer_params ............. None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0} [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] pld_enabled .................. False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] pld_params ................... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] prescale_gradients ........... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] scheduler_name ............... None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] scheduler_params ............. None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] sparse_attention ............. None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] sparse_gradients_enabled ..... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] steps_per_print .............. 10 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] tensorboard_enabled .......... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] tensorboard_job_name ......... DeepSpeedJobName [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] tensorboard_output_path ...... [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] train_batch_size ............. 4 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] train_micro_batch_size_per_gpu 4 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] wall_clock_breakdown ......... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] world_size ................... 1 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] zero_allow_untested_optimizer False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] zero_config .................. { "allgather_bucket_size": 500000000, "allgather_partitions": true, "contiguous_gradients": false, "cpu_offload": false, "cpu_offload_params": false, "cpu_offload_use_pin_memory": "cpu_offload_use_pin_memory", "elastic_checkpoint": true, "gather_fp16_weights_on_model_save": false, "load_from_fp32_weights": true, "max_live_parameters": 1000000000, "max_reuse_distance": 1000000000, "overlap_comm": false, "param_persistence_threshold": 100000, "prefetch_bucket_size": 50000000, "reduce_bucket_size": 500000000, "reduce_scatter": true, "stage": 0, "sub_group_size": 1000000000000 } [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] zero_enabled ................. False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] zero_optimization_stage ...... 0 [2021-06-29 20:20:44,524] [INFO] [config.py:747:print] json = { "amp":{ "enabled":false, "opt_level":"O1" }, "flops_profiler":{ "detailed":true, "enabled":false, "module_depth":-1, "output_file":null, "profile_step":200, "top_modules":1 }, "fp16":{ "enabled":true }, "gradient_accumulation_steps":1, "gradient_clipping":0.5, "train_batch_size":4 } Using /home/valterjordan/.cache/torch_extensions as PyTorch extensions root... Emitting ninja build file /home/valterjordan/.cache/torch_extensions/utils/build.ninja... Building extension module utils... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module utils... Time to load utils op: 0.1738746166229248 seconds [2021-06-29 20:20:45,005] [INFO] [logging.py:60:log_dist] [Rank 0] Saving model checkpoint: dalle-ds-cp/global_step0/mp_rank_00_model_states.pt Traceback (most recent call last): File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 368, in reduce return recipe.apply(tensor) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 205, in apply backend.shape(tensor)) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 150, in reconstruct_from_shape raise EinopsError('Expected {} dimensions, got {}'.format(len(self.input_composite_axes), len(shape))) einops.EinopsError: Expected 2 dimensions, got 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train_dalle.py", line 553, in <module> loss = distr_dalle(text, images, return_loss=True) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 914, in forward loss = self.module(*inputs, **kwargs) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/valterjordan/DALLE-pytorch/dalle_pytorch/dalle_pytorch.py", line 485, in forward image = self.vae.get_codebook_indices(image) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context return func(*args, **kwargs) File "/home/valterjordan/DALLE-pytorch/dalle_pytorch/vae.py", line 199, in get_codebook_indices return rearrange(indices, '(b n) () -> b n', b = b) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 424, in rearrange return reduce(tensor, pattern, reduction='rearrange', **axes_lengths) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 376, in reduce raise EinopsError(message + '\n {}'.format(e)) einops.EinopsError: Error while processing rearrange-reduction pattern "(b n) () -> b n". Input tensor shape: torch.Size([1024]). Additional info: {'b': 4}. Expected 2 dimensions, got 1

closed time in a month

johngore123
IssuesEvent

issue closedlucidrains/DALLE-pytorch

Got error '' Input tensor shape: torch.Size([1024]). Additional info: {'b': 4}. Expected 2 dimensions, got 1'' everything used to work fine now it keeps giving me this error, it only happens on taming.

deepspeed train_dalle.py --local_rank=0 --image_text_folder /home/valterjordan/DALLE-pytorch/datasets/train2017 --truncate_captions --deepspeed --distributed_backend deepspeed --fp16 --taming [2021-06-29 20:20:36,222] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only. [2021-06-29 20:20:36,232] [INFO] [runner.py:358:main] cmd = /home/valterjordan/miniconda3/envs/DALLE-pytorch/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=29500 train_dalle.py --local_rank=0 --image_text_folder /home/valterjordan/DALLE-pytorch/datasets/train2017 --truncate_captions --deepspeed --distributed_backend deepspeed --fp16 --taming

[2021-06-29 20:20:36,528] [INFO] [launch.py:80:main] WORLD INFO DICT: {'localhost': [0]} [2021-06-29 20:20:36,529] [INFO] [launch.py:89:main] nnodes=1, num_local_procs=1, node_rank=0 [2021-06-29 20:20:36,529] [INFO] [launch.py:101:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]}) [2021-06-29 20:20:36,529] [INFO] [launch.py:102:main] dist_world_size=1 [2021-06-29 20:20:36,529] [INFO] [launch.py:105:main] Setting CUDA_VISIBLE_DEVICES=0 /home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/pytorch_lightning/metrics/init.py:44: LightningDeprecationWarning: pytorch_lightning.metrics.* module has been renamed to torchmetrics.* and split off to its own package (https://github.com/PyTorchLightning/metrics) since v1.3 and will be removed in v1.5 "pytorch_lightning.metrics.* module has been renamed to torchmetrics.* and split off to its own package" Using DeepSpeed for distributed execution [2021-06-29 20:20:37,314] [INFO] [distributed.py:47:init_distributed] Initializing torch distributed with backend: nccl using pretrained VAE for encoding images to tokens Working with z of shape (1, 256, 16, 16) = 65536 dimensions. loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth VQLPIPSWithDiscriminator running with hinge loss. Loaded VQGAN from /home/valterjordan/.cache/dalle/vqgan.1024.model.ckpt and /home/valterjordan/.cache/dalle/vqgan.1024.config.yml 118287 image-text pairs found for training wandb: W&B syncing is set to offline in this directory. Run wandb online or set WANDB_MODE=online to enable cloud syncing. [2021-06-29 20:20:43,289] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.13+3352086, git-hash=3352086, git-branch=sparse_triton_support [2021-06-29 20:20:44,498] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 1, parameter_parallel_size: 1 [2021-06-29 20:20:44,513] [INFO] [engine.py:598:_configure_optimizer] Removing param_group that has no 'params'in the client Optimizer [2021-06-29 20:20:44,513] [INFO] [engine.py:602:_configure_optimizer] Using client Optimizer as basic optimizer [2021-06-29 20:20:44,513] [INFO] [engine.py:612:_configure_optimizer] DeepSpeed Basic Optimizer = Adam [2021-06-29 20:20:44,513] [INFO] [logging.py:60:log_dist] [Rank 0] Creating fp16 unfused optimizer with dynamic loss scale [2021-06-29 20:20:44,513] [INFO] [unfused_optimizer.py:36:init] Fused Lamb Legacy : False [2021-06-29 20:20:44,522] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed Final Optimizer = Adam [2021-06-29 20:20:44,522] [INFO] [engine.py:449:_configure_lr_scheduler] DeepSpeed using client LR scheduler [2021-06-29 20:20:44,522] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed LR Scheduler = None [2021-06-29 20:20:44,522] [INFO] [logging.py:60:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0003], mom=[(0.9, 0.999)] [2021-06-29 20:20:44,522] [INFO] [config.py:737:print] DeepSpeedEngine configuration: [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] activation_checkpointing_config { "contiguous_memory_optimization": false, "cpu_checkpointing": false, "number_checkpoints": null, "partition_activations": false, "profile": false, "synchronize_checkpoint_boundary": false } [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] allreduce_always_fp32 ........ False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] amp_enabled .................. False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] amp_params ................... {'opt_level': 'O1'} [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] checkpoint_tag_validation_enabled True [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] checkpoint_tag_validation_fail False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] disable_allgather ............ False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] dump_state ................... False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] dynamic_loss_scale_args ...... None [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] elasticity_enabled ........... False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] flops_profiler_config ........ { "detailed": true, "enabled": false, "module_depth": -1, "profile_step": 200, "top_modules": 1 } [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] fp16_enabled ................. True [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] global_rank .................. 0 [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] gradient_accumulation_steps .. 1 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] gradient_clipping ............ 0.5 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] gradient_predivide_factor .... 1.0 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] initial_dynamic_scale ........ 4294967296 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] loss_scale ................... 0 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] memory_breakdown ............. False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] optimizer_legacy_fusion ...... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] optimizer_name ............... None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] optimizer_params ............. None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0} [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] pld_enabled .................. False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] pld_params ................... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] prescale_gradients ........... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] scheduler_name ............... None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] scheduler_params ............. None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] sparse_attention ............. None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] sparse_gradients_enabled ..... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] steps_per_print .............. 10 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] tensorboard_enabled .......... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] tensorboard_job_name ......... DeepSpeedJobName [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] tensorboard_output_path ...... [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] train_batch_size ............. 4 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] train_micro_batch_size_per_gpu 4 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] wall_clock_breakdown ......... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] world_size ................... 1 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] zero_allow_untested_optimizer False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] zero_config .................. { "allgather_bucket_size": 500000000, "allgather_partitions": true, "contiguous_gradients": false, "cpu_offload": false, "cpu_offload_params": false, "cpu_offload_use_pin_memory": "cpu_offload_use_pin_memory", "elastic_checkpoint": true, "gather_fp16_weights_on_model_save": false, "load_from_fp32_weights": true, "max_live_parameters": 1000000000, "max_reuse_distance": 1000000000, "overlap_comm": false, "param_persistence_threshold": 100000, "prefetch_bucket_size": 50000000, "reduce_bucket_size": 500000000, "reduce_scatter": true, "stage": 0, "sub_group_size": 1000000000000 } [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] zero_enabled ................. False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] zero_optimization_stage ...... 0 [2021-06-29 20:20:44,524] [INFO] [config.py:747:print] json = { "amp":{ "enabled":false, "opt_level":"O1" }, "flops_profiler":{ "detailed":true, "enabled":false, "module_depth":-1, "output_file":null, "profile_step":200, "top_modules":1 }, "fp16":{ "enabled":true }, "gradient_accumulation_steps":1, "gradient_clipping":0.5, "train_batch_size":4 } Using /home/valterjordan/.cache/torch_extensions as PyTorch extensions root... Emitting ninja build file /home/valterjordan/.cache/torch_extensions/utils/build.ninja... Building extension module utils... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module utils... Time to load utils op: 0.1738746166229248 seconds [2021-06-29 20:20:45,005] [INFO] [logging.py:60:log_dist] [Rank 0] Saving model checkpoint: dalle-ds-cp/global_step0/mp_rank_00_model_states.pt Traceback (most recent call last): File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 368, in reduce return recipe.apply(tensor) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 205, in apply backend.shape(tensor)) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 150, in reconstruct_from_shape raise EinopsError('Expected {} dimensions, got {}'.format(len(self.input_composite_axes), len(shape))) einops.EinopsError: Expected 2 dimensions, got 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train_dalle.py", line 553, in <module> loss = distr_dalle(text, images, return_loss=True) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 914, in forward loss = self.module(*inputs, **kwargs) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/valterjordan/DALLE-pytorch/dalle_pytorch/dalle_pytorch.py", line 485, in forward image = self.vae.get_codebook_indices(image) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context return func(*args, **kwargs) File "/home/valterjordan/DALLE-pytorch/dalle_pytorch/vae.py", line 199, in get_codebook_indices return rearrange(indices, '(b n) () -> b n', b = b) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 424, in rearrange return reduce(tensor, pattern, reduction='rearrange', **axes_lengths) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 376, in reduce raise EinopsError(message + '\n {}'.format(e)) einops.EinopsError: Error while processing rearrange-reduction pattern "(b n) () -> b n". Input tensor shape: torch.Size([1024]). Additional info: {'b': 4}. Expected 2 dimensions, got 1

closed time in a month

johngore123

issue openedlucidrains/DALLE-pytorch

Got error '' Input tensor shape: torch.Size([1024]). Additional info: {'b': 4}. Expected 2 dimensions, got 1'' everything used to work fine now it keeps giving me this error, it only happens on taming.

deepspeed train_dalle.py --local_rank=0 --image_text_folder /home/valterjordan/DALLE-pytorch/datasets/train2017 --truncate_captions --deepspeed --distributed_backend deepspeed --fp16 --taming [2021-06-29 20:20:36,222] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only. [2021-06-29 20:20:36,232] [INFO] [runner.py:358:main] cmd = /home/valterjordan/miniconda3/envs/DALLE-pytorch/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=29500 train_dalle.py --local_rank=0 --image_text_folder /home/valterjordan/DALLE-pytorch/datasets/train2017 --truncate_captions --deepspeed --distributed_backend deepspeed --fp16 --taming

[2021-06-29 20:20:36,528] [INFO] [launch.py:80:main] WORLD INFO DICT: {'localhost': [0]} [2021-06-29 20:20:36,529] [INFO] [launch.py:89:main] nnodes=1, num_local_procs=1, node_rank=0 [2021-06-29 20:20:36,529] [INFO] [launch.py:101:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]}) [2021-06-29 20:20:36,529] [INFO] [launch.py:102:main] dist_world_size=1 [2021-06-29 20:20:36,529] [INFO] [launch.py:105:main] Setting CUDA_VISIBLE_DEVICES=0 /home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/pytorch_lightning/metrics/init.py:44: LightningDeprecationWarning: pytorch_lightning.metrics.* module has been renamed to torchmetrics.* and split off to its own package (https://github.com/PyTorchLightning/metrics) since v1.3 and will be removed in v1.5 "pytorch_lightning.metrics.* module has been renamed to torchmetrics.* and split off to its own package" Using DeepSpeed for distributed execution [2021-06-29 20:20:37,314] [INFO] [distributed.py:47:init_distributed] Initializing torch distributed with backend: nccl using pretrained VAE for encoding images to tokens Working with z of shape (1, 256, 16, 16) = 65536 dimensions. loaded pretrained LPIPS loss from taming/modules/autoencoder/lpips/vgg.pth VQLPIPSWithDiscriminator running with hinge loss. Loaded VQGAN from /home/valterjordan/.cache/dalle/vqgan.1024.model.ckpt and /home/valterjordan/.cache/dalle/vqgan.1024.config.yml 118287 image-text pairs found for training wandb: W&B syncing is set to offline in this directory. Run wandb online or set WANDB_MODE=online to enable cloud syncing. [2021-06-29 20:20:43,289] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.13+3352086, git-hash=3352086, git-branch=sparse_triton_support [2021-06-29 20:20:44,498] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 1, parameter_parallel_size: 1 [2021-06-29 20:20:44,513] [INFO] [engine.py:598:_configure_optimizer] Removing param_group that has no 'params'in the client Optimizer [2021-06-29 20:20:44,513] [INFO] [engine.py:602:_configure_optimizer] Using client Optimizer as basic optimizer [2021-06-29 20:20:44,513] [INFO] [engine.py:612:_configure_optimizer] DeepSpeed Basic Optimizer = Adam [2021-06-29 20:20:44,513] [INFO] [logging.py:60:log_dist] [Rank 0] Creating fp16 unfused optimizer with dynamic loss scale [2021-06-29 20:20:44,513] [INFO] [unfused_optimizer.py:36:init] Fused Lamb Legacy : False [2021-06-29 20:20:44,522] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed Final Optimizer = Adam [2021-06-29 20:20:44,522] [INFO] [engine.py:449:_configure_lr_scheduler] DeepSpeed using client LR scheduler [2021-06-29 20:20:44,522] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed LR Scheduler = None [2021-06-29 20:20:44,522] [INFO] [logging.py:60:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0003], mom=[(0.9, 0.999)] [2021-06-29 20:20:44,522] [INFO] [config.py:737:print] DeepSpeedEngine configuration: [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] activation_checkpointing_config { "contiguous_memory_optimization": false, "cpu_checkpointing": false, "number_checkpoints": null, "partition_activations": false, "profile": false, "synchronize_checkpoint_boundary": false } [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] allreduce_always_fp32 ........ False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] amp_enabled .................. False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] amp_params ................... {'opt_level': 'O1'} [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] checkpoint_tag_validation_enabled True [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] checkpoint_tag_validation_fail False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] disable_allgather ............ False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] dump_state ................... False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] dynamic_loss_scale_args ...... None [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] elasticity_enabled ........... False [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] flops_profiler_config ........ { "detailed": true, "enabled": false, "module_depth": -1, "profile_step": 200, "top_modules": 1 } [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] fp16_enabled ................. True [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] global_rank .................. 0 [2021-06-29 20:20:44,522] [INFO] [config.py:741:print] gradient_accumulation_steps .. 1 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] gradient_clipping ............ 0.5 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] gradient_predivide_factor .... 1.0 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] initial_dynamic_scale ........ 4294967296 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] loss_scale ................... 0 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] memory_breakdown ............. False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] optimizer_legacy_fusion ...... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] optimizer_name ............... None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] optimizer_params ............. None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0} [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] pld_enabled .................. False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] pld_params ................... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] prescale_gradients ........... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] scheduler_name ............... None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] scheduler_params ............. None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] sparse_attention ............. None [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] sparse_gradients_enabled ..... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] steps_per_print .............. 10 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] tensorboard_enabled .......... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] tensorboard_job_name ......... DeepSpeedJobName [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] tensorboard_output_path ...... [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] train_batch_size ............. 4 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] train_micro_batch_size_per_gpu 4 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] wall_clock_breakdown ......... False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] world_size ................... 1 [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] zero_allow_untested_optimizer False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] zero_config .................. { "allgather_bucket_size": 500000000, "allgather_partitions": true, "contiguous_gradients": false, "cpu_offload": false, "cpu_offload_params": false, "cpu_offload_use_pin_memory": "cpu_offload_use_pin_memory", "elastic_checkpoint": true, "gather_fp16_weights_on_model_save": false, "load_from_fp32_weights": true, "max_live_parameters": 1000000000, "max_reuse_distance": 1000000000, "overlap_comm": false, "param_persistence_threshold": 100000, "prefetch_bucket_size": 50000000, "reduce_bucket_size": 500000000, "reduce_scatter": true, "stage": 0, "sub_group_size": 1000000000000 } [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] zero_enabled ................. False [2021-06-29 20:20:44,523] [INFO] [config.py:741:print] zero_optimization_stage ...... 0 [2021-06-29 20:20:44,524] [INFO] [config.py:747:print] json = { "amp":{ "enabled":false, "opt_level":"O1" }, "flops_profiler":{ "detailed":true, "enabled":false, "module_depth":-1, "output_file":null, "profile_step":200, "top_modules":1 }, "fp16":{ "enabled":true }, "gradient_accumulation_steps":1, "gradient_clipping":0.5, "train_batch_size":4 } Using /home/valterjordan/.cache/torch_extensions as PyTorch extensions root... Emitting ninja build file /home/valterjordan/.cache/torch_extensions/utils/build.ninja... Building extension module utils... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module utils... Time to load utils op: 0.1738746166229248 seconds [2021-06-29 20:20:45,005] [INFO] [logging.py:60:log_dist] [Rank 0] Saving model checkpoint: dalle-ds-cp/global_step0/mp_rank_00_model_states.pt Traceback (most recent call last): File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 368, in reduce return recipe.apply(tensor) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 205, in apply backend.shape(tensor)) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 150, in reconstruct_from_shape raise EinopsError('Expected {} dimensions, got {}'.format(len(self.input_composite_axes), len(shape))) einops.EinopsError: Expected 2 dimensions, got 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "train_dalle.py", line 553, in <module> loss = distr_dalle(text, images, return_loss=True) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 914, in forward loss = self.module(*inputs, **kwargs) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/home/valterjordan/DALLE-pytorch/dalle_pytorch/dalle_pytorch.py", line 485, in forward image = self.vae.get_codebook_indices(image) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context return func(*args, **kwargs) File "/home/valterjordan/DALLE-pytorch/dalle_pytorch/vae.py", line 199, in get_codebook_indices return rearrange(indices, '(b n) () -> b n', b = b) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 424, in rearrange return reduce(tensor, pattern, reduction='rearrange', **axes_lengths) File "/home/valterjordan/miniconda3/envs/DALLE-pytorch/lib/python3.7/site-packages/einops/einops.py", line 376, in reduce raise EinopsError(message + '\n {}'.format(e)) einops.EinopsError: Error while processing rearrange-reduction pattern "(b n) () -> b n". Input tensor shape: torch.Size([1024]). Additional info: {'b': 4}. Expected 2 dimensions, got 1

created time in a month

issue commentTHUDM/CogView

Got error ''IndexError: tuple index out of range'' running super-res on colab with a tesla v100

Closing forever since problem is solved

johngore123

comment created time in a month

issue closedTHUDM/CogView

Got error ''IndexError: tuple index out of range'' running super-res on colab with a tesla v100

/content/CogView Generate Samples WARNING: No training data specified using world size: 1 and model-parallel size: 1

using dynamic loss scaling initializing model parallel with size 1 initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234 padded vocab (size: 58219) with 21 dummy tokens (new size: 58240) prepare tokenizer done building CogView2 model ... number of parameters on model parallel rank 0: 3928849920 current device: 0 tcmalloc: large alloc 7881007104 bytes == 0x5637e3fb2000 @ 0x7f61e428db6b 0x7f61e42ad379 0x7f6171f1e25e 0x7f6171f1f9d2 0x7f61aff48e7d 0x7f61c0b43120 0x7f61c0781bd9 0x5637152088a8 0x56371527bfd5 0x5637152767ad 0x5637152093ea 0x5637152773b5 0x5637152767ad 0x563715209003 0x563715208b09 0x56371535028d 0x5637152bf1db 0x563715207bb1 0x5637152f8fed 0x56371527b988 0x5637152767ad 0x563715148e2c 0x563715278bb5 0x5637152764ae 0x5637152093ea 0x56371527832a 0x56371520930a 0x5637152773b5 0x56371520930a 0x5637152773b5 0x5637152764ae Load model file pretrained/cogview/cogview-sr/20000/mp_rank_00_model_states.pt Working on No. 0 on 0... Traceback (most recent call last): File "generate_samples.py", line 326, in <module> main() File "generate_samples.py", line 323, in main generate_images_continually(model, args) File "generate_samples.py", line 215, in generate_images_continually for raw_text, seq, output_path in get_context(args, query_template): File "generate_samples.py", line 132, in get_context seq = _parse_and_to_tensor(raw_text, img_size=img_size, query_template=query_template) File "generate_samples.py", line 70, in _parse_and_to_tensor text = query_template.format(*text.split('\t')) IndexError: tuple index out of range /content

closed time in a month

johngore123

issue commentTHUDM/CogView

Got error ''IndexError: tuple index out of range'' running super-res on colab with a tesla v100

Since nobody has gotten it to work.

johngore123

comment created time in a month

issue commentTHUDM/CogView

Got error ''IndexError: tuple index out of range'' running super-res on colab with a tesla v100

Still same error is there any way you could update the readme for an more detailed guide?

johngore123

comment created time in a month

issue commentTHUDM/CogView

Got error ''IndexError: tuple index out of range'' running super-res on colab with a tesla v100

With Chinese text but you get the idea, i think.

johngore123

comment created time in a month

issue commentTHUDM/CogView

Got error ''IndexError: tuple index out of range'' running super-res on colab with a tesla v100

And those are just the default i did replace prompt =''text

johngore123

comment created time in a month

issue commentTHUDM/CogView

Got error ''IndexError: tuple index out of range'' running super-res on colab with a tesla v100

#@title Super resolution (WIP) if you know how to implement this contact aze#0001 on discord #@markdown this requires the cogview-sr model to be downloaded (look at the 3rd cell)

#@markdown can be 0.jpg, 1.jpg, 2.jpg ... etc depends on your batch_size image = "0.jpg" #@param {type:"string"}

prompt = "text\t/content/CogView/samples/" + image

%cd /content/CogView !sh scripts/super_resolution.sh %cd /content

一个漂亮的女人走在街上\t/content/CogView/samples/0.jpg

johngore123

comment created time in a month

issue commentTHUDM/CogView

Got error ''IndexError: tuple index out of range'' running super-res on colab with a tesla v100

Could need some more documentation.

johngore123

comment created time in a month

issue commentTHUDM/CogView

Got error ''IndexError: tuple index out of range'' running super-res on colab with a tesla v100

Issue still happens even with all those things.

johngore123

comment created time in a month

IssuesEvent

issue closedTHUDM/CogView

Got error ''IndexError: tuple index out of range'' running super-res on colab with a tesla v100

/content/CogView Generate Samples WARNING: No training data specified using world size: 1 and model-parallel size: 1

using dynamic loss scaling initializing model parallel with size 1 initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234 padded vocab (size: 58219) with 21 dummy tokens (new size: 58240) prepare tokenizer done building CogView2 model ... number of parameters on model parallel rank 0: 3928849920 current device: 0 tcmalloc: large alloc 7881007104 bytes == 0x5637e3fb2000 @ 0x7f61e428db6b 0x7f61e42ad379 0x7f6171f1e25e 0x7f6171f1f9d2 0x7f61aff48e7d 0x7f61c0b43120 0x7f61c0781bd9 0x5637152088a8 0x56371527bfd5 0x5637152767ad 0x5637152093ea 0x5637152773b5 0x5637152767ad 0x563715209003 0x563715208b09 0x56371535028d 0x5637152bf1db 0x563715207bb1 0x5637152f8fed 0x56371527b988 0x5637152767ad 0x563715148e2c 0x563715278bb5 0x5637152764ae 0x5637152093ea 0x56371527832a 0x56371520930a 0x5637152773b5 0x56371520930a 0x5637152773b5 0x5637152764ae Load model file pretrained/cogview/cogview-sr/20000/mp_rank_00_model_states.pt Working on No. 0 on 0... Traceback (most recent call last): File "generate_samples.py", line 326, in <module> main() File "generate_samples.py", line 323, in main generate_images_continually(model, args) File "generate_samples.py", line 215, in generate_images_continually for raw_text, seq, output_path in get_context(args, query_template): File "generate_samples.py", line 132, in get_context seq = _parse_and_to_tensor(raw_text, img_size=img_size, query_template=query_template) File "generate_samples.py", line 70, in _parse_and_to_tensor text = query_template.format(*text.split('\t')) IndexError: tuple index out of range /content

closed time in a month

johngore123