1. In app.py

    if sys.argv[1] == 'index':
        workspace = os.environ['WORKDIR']
        if os.path.exists(workspace):
            print(f'\n +---------------------------------------------------------------------------------+ \
                    \n |                                   🤖🤖🤖                                        | \
                    \n | The directory {workspace} already exists. Please remove it before indexing again. | \
                    \n |                                   🤖🤖🤖                                        | \
                    \n +---------------------------------------------------------------------------------+')
        config()
        index()

shows error:

(py37-lyrics) sz@air:/mnt/d/coding/jina-ai/examples/multires-lyrics-search$ python3.7 app.py index
Traceback (most recent call last):
  File "app.py", line 70, in <module>
    if (os.environ['WORKDIR']):
  File "/usr/lib/python3.7/os.py", line 681, in __getitem__
    raise KeyError(key) from None
KeyError: 'WORKDIR'

I think the logic here is not quite right if there is no WORKDIR at the beginning.

2. By pass the 1) issue as

    if sys.argv[1] == 'index':
        config()
        index()

still get errors:

(py37-lyrics) sz@air:/mnt/d/coding/jina-ai/examples/multires-lyrics-search$ python3.7 app.py index
           Flow@5397[S]:successfully built Flow from a yaml config
        crafter@5417[I]:post initiating, this may take some time...
        crafter@5417[I]:post initiating, this may take some time takes 0 seconds (0.00s)
        crafter@5417[S]:successfully built Sentencizer from a yaml config
        crafter@5417[I]:setting up sockets...
        crafter@5417[I]:input tcp://0.0.0.0:54983 (SUB_CONNECT)          output tcp://0.0.0.0:47037 (PUSH_CONNECT)     control over tcp://0.0.0.0:47793 (PAIR_BIND)
        crafter@5417[S]:ready and listening
        encoder@5422[I]:post initiating, this may take some time...
        encoder@5422[I]:post initiating, this may take some time takes 0 seconds (0.00s)
        encoder@5422[S]:successfully built BaseExecutor from a yaml config
        encoder@5422[I]:setting up sockets...
        encoder@5422[I]:input tcp://0.0.0.0:47037 (PULL_BIND)    output tcp://0.0.0.0:58657 (ROUTER_BIND)      control over tcp://0.0.0.0:45535 (PAIR_BIND)
        encoder@5422[S]:ready and listening
        encoder@5427[I]:post initiating, this may take some time...
        encoder@5427[I]:post initiating, this may take some time takes 0 seconds (0.00s)
        encoder@5427[S]:successfully built BaseExecutor from a yaml config
        encoder@5427[I]:setting up sockets...
        encoder@5427[I]:input tcp://0.0.0.0:42049 (PULL_BIND)    output tcp://0.0.0.0:39111 (PUSH_CONNECT)     control over tcp://0.0.0.0:43847 (PAIR_BIND)
        encoder@5427[S]:ready and listening
        encoder@5432[I]:post initiating, this may take some time...
        encoder@5432[I]:post initiating, this may take some time takes 0 seconds (0.00s)
        encoder@5432[S]:successfully built TransformerTorchEncoder from a yaml config
        encoder@5432[I]:setting up sockets...
        encoder@5432[I]:input tcp://0.0.0.0:58657 (DEALER_CONNECT)       output tcp://0.0.0.0:42049 (PUSH_CONNECT)     control over tcp://0.0.0.0:59905 (PAIR_BIND)
        encoder@5432[S]:ready and listening
        encoder@5422[I]:received ControlRequest from encoder▸⚐
        encoder@5437[I]:post initiating, this may take some time...
        encoder@5437[I]:post initiating, this may take some time takes 0 seconds (0.00s)
        encoder@5437[S]:successfully built TransformerTorchEncoder from a yaml config
        encoder@5437[I]:setting up sockets...
        encoder@5437[I]:input tcp://0.0.0.0:58657 (DEALER_CONNECT)       output tcp://0.0.0.0:42049 (PUSH_CONNECT)     control over tcp://0.0.0.0:33847 (PAIR_BIND)
        encoder@5437[S]:ready and listening
        encoder@5422[I]:received ControlRequest from encoder▸⚐
      chunk_idx@5442[I]:post initiating, this may take some time...
      chunk_idx@5442[I]:post initiating, this may take some time takes 0 seconds (0.00s)
      chunk_idx@5442[S]:successfully built BaseExecutor from a yaml config
      chunk_idx@5442[I]:setting up sockets...
      chunk_idx@5442[I]:input tcp://0.0.0.0:39111 (PULL_BIND)    output tcp://0.0.0.0:33097 (ROUTER_BIND)      control over tcp://0.0.0.0:32853 (PAIR_BIND)
      chunk_idx@5442[S]:ready and listening
      chunk_idx@5447[I]:post initiating, this may take some time...
      chunk_idx@5447[I]:post initiating, this may take some time takes 0 seconds (0.00s)
      chunk_idx@5447[S]:successfully built BaseExecutor from a yaml config
      chunk_idx@5447[I]:setting up sockets...
      chunk_idx@5447[I]:input tcp://0.0.0.0:43389 (PULL_BIND)    output tcp://0.0.0.0:57583 (PUSH_CONNECT)     control over tcp://0.0.0.0:37655 (PAIR_BIND)
      chunk_idx@5447[S]:ready and listening
      chunk_idx@5452[I]:post initiating, this may take some time...
      chunk_idx@5452[I]:post initiating, this may take some time takes 0 seconds (0.00s)
      chunk_idx@5452[C]:unknown exception: ValueError('workspace=$JINA_WORKSPACE is not expandable or badly referred')
Traceback (most recent call last):
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/peapods/pea.py", line 348, in run
    self.loop_body()
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/peapods/pea.py", line 329, in loop_body
    self.load_executor()
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/peapods/pea.py", line 200, in load_executor
    self.args.separated_workspace, self.args.pea_id)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/executors/__init__.py", line 433, in load_config
    return yaml.load(tmp_s)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/main.py", line 343, in load
    return constructor.get_single_data()
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 113, in get_single_data
    return self.construct_document(node)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 118, in construct_document
    data = self.construct_object(node)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 146, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 181, in construct_non_recursive_object
    data = constructor(self, node)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/executors/compound.py", line 350, in from_yaml
    obj, data, from_dump = super()._get_instance_from_yaml(constructor, node)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/executors/__init__.py", line 479, in _get_instance_from_yaml
    constructor, node, deep=True)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 440, in construct_mapping
    return BaseConstructor.construct_mapping(self, node, deep=deep)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 255, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 146, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 188, in construct_non_recursive_object
    for _dummy in generator:
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 717, in construct_yaml_seq
    data.extend(self.construct_sequence(node))
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 211, in construct_sequence
    return [self.construct_object(child, deep=deep) for child in node.value]
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 211, in <listcomp>
    return [self.construct_object(child, deep=deep) for child in node.value]
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 146, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/ruamel/yaml/constructor.py", line 181, in construct_non_recursive_object
    data = constructor(self, node)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/executors/__init__.py", line 474, in from_yaml
    return cls._get_instance_from_yaml(constructor, node)[0]
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/executors/__init__.py", line 507, in _get_instance_from_yaml
    obj = cls(**data.get('with', {}), metas=data.get('metas', {}), requests=data.get('requests', {}))
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/executors/__init__.py", line 59, in __call__
    getattr(obj, '_post_init_wrapper', lambda *x: None)(m, r)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/executors/__init__.py", line 162, in _post_init_wrapper
    self._fill_metas(_metas)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/executors/__init__.py", line 224, in _fill_metas
    raise ValueError(f'{k}={v} is not expandable or badly referred')
ValueError: workspace=$JINA_WORKSPACE is not expandable or badly referred
      chunk_idx@5452[S]:terminated
      chunk_idx@5397[C]:fail to start <class 'jina.peapods.pea.BasePea'> with name chunk_idx-1, this often means the executor used in the pod is not valid
Traceback (most recent call last):
  File "app.py", line 79, in <module>
    index()
  File "app.py", line 44, in index
    with f:
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/flow/__init__.py", line 451, in __enter__
    return self.start()
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/flow/__init__.py", line 507, in start
    self.enter_context(v)
  File "/usr/lib/python3.7/contextlib.py", line 427, in enter_context
    result = _cm_type.__enter__(cm)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/peapods/pod.py", line 254, in __enter__
    return self.start()
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/peapods/pod.py", line 389, in start
    return super().start()
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/peapods/pod.py", line 224, in start
    self.enter_context(p)
  File "/usr/lib/python3.7/contextlib.py", line 427, in enter_context
    result = _cm_type.__enter__(cm)
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/peapods/pea.py", line 425, in __enter__
    return self.start()
  File "/home/sz/py37-lyrics/lib/python3.7/site-packages/jina/peapods/pea.py", line 418, in start
    raise PeaFailToStart
jina.excepts.PeaFailToStart

'workspace=$JINA_WORKSPACE is not expandable or badly referred' What does this mean?

It is needed :(

@JoanFM It's not complaining

No comments from my side as I am not familiar with remote functionality

Good move! This part of the documentation is really out-of-date.

Right now, there are two functions for adding a chunk/match to the document. I'd suggest removing add().

There should be one-- and preferably only one --obvious way to do it. The Zen of Python

with Document() as d:
    c1 = d.chunks.append()
    c1.id = f'1:0>16'
    c2 = d.chunks.add()
    c2.id = f'2:0>16'

Codecov Report

Merging #1346 (56a0550) into master (f8db230) will increase coverage by 0.01%. The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1346      +/-   ##
==========================================
+ Coverage   83.27%   83.28%   +0.01%     
==========================================
  Files         103      103              
  Lines        6744     6744              
==========================================
+ Hits         5616     5617       +1     
+ Misses       1128     1127       -1     

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update f8db230...390f062. Read the comment docs.

--allow-spawn has been removed in #1276 , while it still exists in the documentation, run-remote-pod-jina.md

1. Removed index remote 2 from tests, since it testing the delete args --allow-spawn.
2. Refactor index remote test.
3. Remove JinaTestCase from __init__, we're not using unittest anymore.

@bio-howard you can continue with the PR now, please replace jina_pb2. DocumentProto with Document, import from jina.type

hi @rabbits99 would you like to continue with this PR? it has been 2 weeks. Please check commit guidelines #553 to update your commit message, and sign the CLA.

To me as an application developer, it would be useful to load ndarray images into the flow and apply image crafters on these images.

I will close this issue, we discussed in the PR that this functionality should not be part of ImageReader. The Flow already accepts inputing np arrays and working with them

@JoanFM @theUnkownName shall we proceed with this PR? looks good now

Sure

i don't see the point of pouting the constructor here.

@JoanFM @theUnkownName shall we proceed with this PR? looks good now

Yes it was just a test I will change in there.

I think I was wrong it may be actually a bug at batching level. I am howevrr not sure we can assume it returns an np.ndarray. batching decorator has too many things, if you find a pattern to abstract in another function it can also be good

the previous image version is 0.0.3

Due to the release of jina core v0.8.1, this draft PR is created in order to trigger an automatic build & push of the module

Automatic build successful. Image has been built and deployed.

Automatic build successful. Image has been built and deployed.