If you are wondering where the data of this site comes from, please visit https://api.github.com/users/BastinJafari/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

Bastin BastinJafari

repos

9

gists

0

followers

6

following

1

BastinJafari/Mastering-Ethereum-for-5.0 1

Updated source code for the book "Mastering Ethereum: Building Smart Contracts and DApps"

BastinJafari/weather-app 1

BastinJafari/dashboard 0

Interactive UI for analyzing logs, designing flows and viewing Hub images

BastinJafari/ModernUO 0

ModernUO - Ultima Online Server Emulator for the modern era

BastinJafari/react-flow-chart 0

A flexible, stateless, declarative flow chart library for react.

BastinJafari/tinpriv-node-backend 0

Backend for Tinpriv App in Node.js

BastinJafari/tinypriv-aws-backend 0

backend for tinypriv.com on aws

BastinJafari/tinypriv-frontend 0

Frontend for

BastinJafari/Visualisation 0

Visualisation of distributed algorithms

issue closedjina-ai/jina

AsyncFlow | TypeError: object async_generator can't be used in 'await' expression

The Bug Following error is raised when awaiting the index() method of the AsyncFlow using the latest Jina release (1.0.7): TypeError: object async_generator can't be used in 'await' expression

Reproduce

from jina import AsyncFlow, Document

async def index_helper(input_fn):
    flow = AsyncFlow.load_config(path)
    with flow as f:
        await f.index(input_fn)  # <-- PROGRAMM FAILS HERE

When I downgrade jina to 0.9 or I remove the await before the f.index everything works fine.

closed time in 4 hours

push eventjina-ai/jina

Maximilian Werk

commit sha a151006980b3a6f0065b3e197062f6654c77a74e

refactor: added fast traversal with structure (#1950) * refactor: added fast traversal with structure * fix: document set requires sequences * test: fix vector index and vector search * refactor: revert to old interface * refactor: kv search uses set recursive mixin * test: completed traversal * feat: almost all driver adapted * refactor: completely removed recursivemixin * refactor: moved docs property to base driver * fix: repaired tests * fix: docstrings * refactor: added traversal per path * docs: fix docstring linter errors * fix: random ports are now not double sampled * fix: port type * fix: test and sync ci and cd again * test: moved port retry to test fixture * test: fixed random port for daemon * fix: final cleanup * test: make fixture more elegant * refactor: fixed mock related imports Co-authored-by: cristian mtr <[email protected]>

commit sha 2f51048c3b2a1c7fb3cf4fe7c4d0fea19581fe5c

chore(style): reformatted by jina-dev-bot

commit sha 4b6af96dec04ac17a9375f44c73ff72ac13c36ec

chore(contributor): update contributors

commit sha cb40b44f05212dbf69f8ef40792d094e51553048

ci: include docstr linter (#2045)

commit sha 3af29051e8e9e35eca5c76c52f86e60df515c834

feat(binarypb): delete on dump (#2102)

Maximilian Werk

commit sha 9bbb0769b0474ddb5a0682b518f41f7e9643ff43

fix: expose env variable for workspace (#2114)

commit sha 7169fb56ad2fa3a919c0225e35fdb0ed08b910e4

fix: fix traversal_path, change from c to r (#2116)

commit sha 640daf4d389768be216dac4125ef4e837ee65d23

ci: add black (#2036) * ci: add black * ci: add git blame

commit sha e01e57df00deda8ea7bbda1f0a26ba25c60782a6

chore(contributor): update contributors

commit sha dc2be2f009b8e82be8241363efd71ce3f32cbf84

ci: reenable docstrings lint (#2118)

commit sha c258e4aa22495d3809ecbcb0ee9966938ccdbfe5

docs: update black docs and sha (#2117)

commit sha 7dd876d0a1fbfca3818c13a68521b80e43a1c617

chore(contributor): update contributors

Florian Hönicke

commit sha 80e81dae07fb0dc33ac71b8b408a2c4c1feab80a

refactor: merge master

push time in 4 hours

Pull request review commentjina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

-from typing import Callable, Dict, Any, Optional, List, Union+from typing import Dict, Any, Optional, List, Union++from enum import Enum+from datetime import datetime+from collections import defaultdict  from pydantic import Field, BaseModel, create_model+from google.protobuf.descriptor import Descriptor, FieldDescriptor+from google.protobuf.pyext.cpp_message import GeneratedProtocolMessageType  from jina.enums import DataInputType from jina.types.document import Document from jina.parsers import set_client_cli_parser-from jina.proto.jina_pb2 import DocumentProto, QueryLangProto+from jina.proto.jina_pb2 import (+    DenseNdArrayProto,+    NdArrayProto,+    SparseNdArrayProto,+    NamedScoreProto,+    DocumentProto,+    RouteProto,+    EnvelopeProto,+    StatusProto,+    MessageProto,+    RequestProto,+    QueryLangProto,+)+++DEFAULT_REQUEST_SIZE = set_client_cli_parser().parse_args([]).request_size+PROTO_TO_PYDANTIC_MODELS = {}+PROTOBUF_TO_PYTHON_TYPE = {+    FieldDescriptor.TYPE_INT32: int,+    FieldDescriptor.TYPE_INT64: int,+    FieldDescriptor.TYPE_UINT32: int,+    FieldDescriptor.TYPE_UINT64: int,+    FieldDescriptor.TYPE_SINT32: int,+    FieldDescriptor.TYPE_SINT64: int,+    FieldDescriptor.TYPE_BOOL: bool,+    FieldDescriptor.TYPE_FLOAT: float,+    FieldDescriptor.TYPE_DOUBLE: float,+    FieldDescriptor.TYPE_FIXED32: float,+    FieldDescriptor.TYPE_FIXED64: float,+    FieldDescriptor.TYPE_SFIXED32: float,+    FieldDescriptor.TYPE_SFIXED64: float,+    FieldDescriptor.TYPE_BYTES: bytes,+    FieldDescriptor.TYPE_STRING: str,+    FieldDescriptor.TYPE_ENUM: Enum,+    FieldDescriptor.TYPE_MESSAGE: None,+}+++def protobuf_to_pydantic_model(+    protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+) -> BaseModel:+    """+    Converts Protobuf messages to Pydantic model for jsonschema creation/validattion++    ..note:: Model gets assigned in the global dict :data:PROTO_TO_PYDANTIC_MODELS++    :param protobuf_model: *Proto message from proto file+    :type protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+    :return: Pydantic model+    :rtype: BaseModel+    """++    all_fields = {}+    oneof_fields = defaultdict(list)++    if isinstance(protobuf_model, Descriptor):+        model_name = protobuf_model.name+        protobuf_fields = protobuf_model.fields+    elif isinstance(protobuf_model, GeneratedProtocolMessageType):+        model_name = protobuf_model.DESCRIPTOR.name+        protobuf_fields = protobuf_model.DESCRIPTOR.fields++    if model_name.endswith('Proto') and model_name in PROTO_TO_PYDANTIC_MODELS:+        return PROTO_TO_PYDANTIC_MODELS[model_name]++    for f in protobuf_fields:+        field_type = PROTOBUF_TO_PYTHON_TYPE[f.type]+        default_value = f.default_value++        if f.containing_oneof:+            # Proto Field type: oneof+            # NOTE: oneof fields are handled as a post-processing step+            oneof_fields[f.containing_oneof.name].append(f.name)++        if field_type is Enum:+            # Proto Field Type: enum+            enum_dict = {}+            for enum_field in f.enum_type.values:+                enum_dict[enum_field.name] = enum_field.number+            field_type = Enum(f.enum_type.name, enum_dict)++        if f.message_type:++            if f.message_type.name == 'Struct':+                # Proto Field Type: google.protobuf.Struct+                field_type = Dict+                default_value = {}++            elif f.message_type.name == 'Timestamp':+                # Proto Field Type: google.protobuf.Timestamp+                field_type = datetime+                default_value = datetime.now()++            elif f.message_type.name.endswith('Proto'):

can we at least centralize the logic somewhere in types, so that it is kept in a single place

deepankarm

comment created time in 6 hours

Pull request review commentjina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

-from typing import Callable, Dict, Any, Optional, List, Union+from typing import Dict, Any, Optional, List, Union++from enum import Enum+from datetime import datetime+from collections import defaultdict  from pydantic import Field, BaseModel, create_model+from google.protobuf.descriptor import Descriptor, FieldDescriptor+from google.protobuf.pyext.cpp_message import GeneratedProtocolMessageType  from jina.enums import DataInputType from jina.types.document import Document from jina.parsers import set_client_cli_parser-from jina.proto.jina_pb2 import DocumentProto, QueryLangProto+from jina.proto.jina_pb2 import (+    DenseNdArrayProto,+    NdArrayProto,+    SparseNdArrayProto,+    NamedScoreProto,+    DocumentProto,+    RouteProto,+    EnvelopeProto,+    StatusProto,+    MessageProto,+    RequestProto,+    QueryLangProto,+)+++DEFAULT_REQUEST_SIZE = set_client_cli_parser().parse_args([]).request_size+PROTO_TO_PYDANTIC_MODELS = {}+PROTOBUF_TO_PYTHON_TYPE = {+    FieldDescriptor.TYPE_INT32: int,+    FieldDescriptor.TYPE_INT64: int,+    FieldDescriptor.TYPE_UINT32: int,+    FieldDescriptor.TYPE_UINT64: int,+    FieldDescriptor.TYPE_SINT32: int,+    FieldDescriptor.TYPE_SINT64: int,+    FieldDescriptor.TYPE_BOOL: bool,+    FieldDescriptor.TYPE_FLOAT: float,+    FieldDescriptor.TYPE_DOUBLE: float,+    FieldDescriptor.TYPE_FIXED32: float,+    FieldDescriptor.TYPE_FIXED64: float,+    FieldDescriptor.TYPE_SFIXED32: float,+    FieldDescriptor.TYPE_SFIXED64: float,+    FieldDescriptor.TYPE_BYTES: bytes,+    FieldDescriptor.TYPE_STRING: str,+    FieldDescriptor.TYPE_ENUM: Enum,+    FieldDescriptor.TYPE_MESSAGE: None,+}+++def protobuf_to_pydantic_model(+    protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+) -> BaseModel:+    """+    Converts Protobuf messages to Pydantic model for jsonschema creation/validattion++    ..note:: Model gets assigned in the global dict :data:PROTO_TO_PYDANTIC_MODELS++    :param protobuf_model: *Proto message from proto file+    :type protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+    :return: Pydantic model+    :rtype: BaseModel+    """++    all_fields = {}+    oneof_fields = defaultdict(list)++    if isinstance(protobuf_model, Descriptor):+        model_name = protobuf_model.name+        protobuf_fields = protobuf_model.fields+    elif isinstance(protobuf_model, GeneratedProtocolMessageType):+        model_name = protobuf_model.DESCRIPTOR.name+        protobuf_fields = protobuf_model.DESCRIPTOR.fields++    if model_name.endswith('Proto') and model_name in PROTO_TO_PYDANTIC_MODELS:+        return PROTO_TO_PYDANTIC_MODELS[model_name]++    for f in protobuf_fields:+        field_type = PROTOBUF_TO_PYTHON_TYPE[f.type]+        default_value = f.default_value++        if f.containing_oneof:+            # Proto Field type: oneof+            # NOTE: oneof fields are handled as a post-processing step+            oneof_fields[f.containing_oneof.name].append(f.name)++        if field_type is Enum:+            # Proto Field Type: enum+            enum_dict = {}+            for enum_field in f.enum_type.values:+                enum_dict[enum_field.name] = enum_field.number+            field_type = Enum(f.enum_type.name, enum_dict)++        if f.message_type:++            if f.message_type.name == 'Struct':+                # Proto Field Type: google.protobuf.Struct+                field_type = Dict+                default_value = {}++            elif f.message_type.name == 'Timestamp':+                # Proto Field Type: google.protobuf.Timestamp+                field_type = datetime+                default_value = datetime.now()++            elif f.message_type.name.endswith('Proto'):

I couldn't find a better way to validate MessageDescriptor objects in protobuf except for the name field. It is true that a simple else would also work, but that brings issues if we introduce more predefined modules in jina.proto.

deepankarm

comment created time in 6 hours

Pull request review commentjina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

-from typing import Callable, Dict, Any, Optional, List, Union+from typing import Dict, Any, Optional, List, Union++from enum import Enum+from datetime import datetime+from collections import defaultdict  from pydantic import Field, BaseModel, create_model+from google.protobuf.descriptor import Descriptor, FieldDescriptor+from google.protobuf.pyext.cpp_message import GeneratedProtocolMessageType  from jina.enums import DataInputType from jina.types.document import Document from jina.parsers import set_client_cli_parser-from jina.proto.jina_pb2 import DocumentProto, QueryLangProto+from jina.proto.jina_pb2 import (+    DenseNdArrayProto,+    NdArrayProto,+    SparseNdArrayProto,+    NamedScoreProto,+    DocumentProto,+    RouteProto,+    EnvelopeProto,+    StatusProto,+    MessageProto,+    RequestProto,+    QueryLangProto,+)+++DEFAULT_REQUEST_SIZE = set_client_cli_parser().parse_args([]).request_size+PROTO_TO_PYDANTIC_MODELS = {}+PROTOBUF_TO_PYTHON_TYPE = {+    FieldDescriptor.TYPE_INT32: int,+    FieldDescriptor.TYPE_INT64: int,+    FieldDescriptor.TYPE_UINT32: int,+    FieldDescriptor.TYPE_UINT64: int,+    FieldDescriptor.TYPE_SINT32: int,+    FieldDescriptor.TYPE_SINT64: int,+    FieldDescriptor.TYPE_BOOL: bool,+    FieldDescriptor.TYPE_FLOAT: float,+    FieldDescriptor.TYPE_DOUBLE: float,+    FieldDescriptor.TYPE_FIXED32: float,+    FieldDescriptor.TYPE_FIXED64: float,+    FieldDescriptor.TYPE_SFIXED32: float,+    FieldDescriptor.TYPE_SFIXED64: float,+    FieldDescriptor.TYPE_BYTES: bytes,+    FieldDescriptor.TYPE_STRING: str,+    FieldDescriptor.TYPE_ENUM: Enum,+    FieldDescriptor.TYPE_MESSAGE: None,+}+++def protobuf_to_pydantic_model(+    protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+) -> BaseModel:+    """+    Converts Protobuf messages to Pydantic model for jsonschema creation/validattion++    ..note:: Model gets assigned in the global dict :data:PROTO_TO_PYDANTIC_MODELS++    :param protobuf_model: *Proto message from proto file+    :type protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+    :return: Pydantic model+    :rtype: BaseModel+    """++    all_fields = {}+    oneof_fields = defaultdict(list)++    if isinstance(protobuf_model, Descriptor):+        model_name = protobuf_model.name+        protobuf_fields = protobuf_model.fields+    elif isinstance(protobuf_model, GeneratedProtocolMessageType):+        model_name = protobuf_model.DESCRIPTOR.name+        protobuf_fields = protobuf_model.DESCRIPTOR.fields++    if model_name.endswith('Proto') and model_name in PROTO_TO_PYDANTIC_MODELS:+        return PROTO_TO_PYDANTIC_MODELS[model_name]++    for f in protobuf_fields:+        field_type = PROTOBUF_TO_PYTHON_TYPE[f.type]+        default_value = f.default_value++        if f.containing_oneof:+            # Proto Field type: oneof+            # NOTE: oneof fields are handled as a post-processing step+            oneof_fields[f.containing_oneof.name].append(f.name)++        if field_type is Enum:+            # Proto Field Type: enum+            enum_dict = {}+            for enum_field in f.enum_type.values:+                enum_dict[enum_field.name] = enum_field.number+            field_type = Enum(f.enum_type.name, enum_dict)++        if f.message_type:++            if f.message_type.name == 'Struct':+                # Proto Field Type: google.protobuf.Struct+                field_type = Dict+                default_value = {}++            elif f.message_type.name == 'Timestamp':+                # Proto Field Type: google.protobuf.Timestamp+                field_type = datetime+                default_value = datetime.now()

Timestamp field needs further polishing. google.protobuf.Timestamp describes time in seconds & nanos, where datetime.now() is in a different format. Since this schema is only exposed to the users for viewing, for now, I would prefer to keep it as-is for now and upgrade as and when required.

deepankarm

comment created time in 7 hours

Pull request review commentjina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

-from typing import Callable, Dict, Any, Optional, List, Union+from typing import Dict, Any, Optional, List, Union++from enum import Enum+from datetime import datetime+from collections import defaultdict  from pydantic import Field, BaseModel, create_model+from google.protobuf.descriptor import Descriptor, FieldDescriptor+from google.protobuf.pyext.cpp_message import GeneratedProtocolMessageType  from jina.enums import DataInputType from jina.types.document import Document from jina.parsers import set_client_cli_parser-from jina.proto.jina_pb2 import DocumentProto, QueryLangProto+from jina.proto.jina_pb2 import (+    DenseNdArrayProto,+    NdArrayProto,+    SparseNdArrayProto,+    NamedScoreProto,+    DocumentProto,+    RouteProto,+    EnvelopeProto,+    StatusProto,+    MessageProto,+    RequestProto,+    QueryLangProto,+)+++DEFAULT_REQUEST_SIZE = set_client_cli_parser().parse_args([]).request_size+PROTO_TO_PYDANTIC_MODELS = {}+PROTOBUF_TO_PYTHON_TYPE = {+    FieldDescriptor.TYPE_INT32: int,+    FieldDescriptor.TYPE_INT64: int,+    FieldDescriptor.TYPE_UINT32: int,+    FieldDescriptor.TYPE_UINT64: int,+    FieldDescriptor.TYPE_SINT32: int,+    FieldDescriptor.TYPE_SINT64: int,+    FieldDescriptor.TYPE_BOOL: bool,+    FieldDescriptor.TYPE_FLOAT: float,+    FieldDescriptor.TYPE_DOUBLE: float,+    FieldDescriptor.TYPE_FIXED32: float,+    FieldDescriptor.TYPE_FIXED64: float,+    FieldDescriptor.TYPE_SFIXED32: float,+    FieldDescriptor.TYPE_SFIXED64: float,+    FieldDescriptor.TYPE_BYTES: bytes,+    FieldDescriptor.TYPE_STRING: str,+    FieldDescriptor.TYPE_ENUM: Enum,+    FieldDescriptor.TYPE_MESSAGE: None,+}+++def protobuf_to_pydantic_model(+    protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+) -> BaseModel:+    """+    Converts Protobuf messages to Pydantic model for jsonschema creation/validattion++    ..note:: Model gets assigned in the global dict :data:PROTO_TO_PYDANTIC_MODELS++    :param protobuf_model: *Proto message from proto file+    :type protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+    :return: Pydantic model+    :rtype: BaseModel+    """++    all_fields = {}+    oneof_fields = defaultdict(list)++    if isinstance(protobuf_model, Descriptor):+        model_name = protobuf_model.name+        protobuf_fields = protobuf_model.fields+    elif isinstance(protobuf_model, GeneratedProtocolMessageType):+        model_name = protobuf_model.DESCRIPTOR.name+        protobuf_fields = protobuf_model.DESCRIPTOR.fields++    if model_name.endswith('Proto') and model_name in PROTO_TO_PYDANTIC_MODELS:+        return PROTO_TO_PYDANTIC_MODELS[model_name]

This helps us in 2 ways

Keeping this global variable adds all schemas to openapi docs. Hence, auto-rendering of all Jina protos in jsonschema for both swagger & redoc
Easier handling of recursive schemas.

deepankarm

comment created time in 7 hours

Pull request review commentjina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

-from typing import Callable, Dict, Any, Optional, List, Union+from typing import Dict, Any, Optional, List, Union++from enum import Enum+from datetime import datetime+from collections import defaultdict  from pydantic import Field, BaseModel, create_model+from google.protobuf.descriptor import Descriptor, FieldDescriptor+from google.protobuf.pyext.cpp_message import GeneratedProtocolMessageType  from jina.enums import DataInputType from jina.types.document import Document from jina.parsers import set_client_cli_parser-from jina.proto.jina_pb2 import DocumentProto, QueryLangProto+from jina.proto.jina_pb2 import (+    DenseNdArrayProto,+    NdArrayProto,+    SparseNdArrayProto,+    NamedScoreProto,+    DocumentProto,+    RouteProto,+    EnvelopeProto,+    StatusProto,+    MessageProto,+    RequestProto,+    QueryLangProto,+)+++DEFAULT_REQUEST_SIZE = set_client_cli_parser().parse_args([]).request_size+PROTO_TO_PYDANTIC_MODELS = {}+PROTOBUF_TO_PYTHON_TYPE = {+    FieldDescriptor.TYPE_INT32: int,+    FieldDescriptor.TYPE_INT64: int,+    FieldDescriptor.TYPE_UINT32: int,+    FieldDescriptor.TYPE_UINT64: int,+    FieldDescriptor.TYPE_SINT32: int,+    FieldDescriptor.TYPE_SINT64: int,+    FieldDescriptor.TYPE_BOOL: bool,+    FieldDescriptor.TYPE_FLOAT: float,+    FieldDescriptor.TYPE_DOUBLE: float,+    FieldDescriptor.TYPE_FIXED32: float,+    FieldDescriptor.TYPE_FIXED64: float,+    FieldDescriptor.TYPE_SFIXED32: float,+    FieldDescriptor.TYPE_SFIXED64: float,+    FieldDescriptor.TYPE_BYTES: bytes,+    FieldDescriptor.TYPE_STRING: str,+    FieldDescriptor.TYPE_ENUM: Enum,+    FieldDescriptor.TYPE_MESSAGE: None,+}+++def protobuf_to_pydantic_model(+    protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+) -> BaseModel:+    """+    Converts Protobuf messages to Pydantic model for jsonschema creation/validattion++    ..note:: Model gets assigned in the global dict :data:PROTO_TO_PYDANTIC_MODELS++    :param protobuf_model: *Proto message from proto file+    :type protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+    :return: Pydantic model+    :rtype: BaseModel+    """++    all_fields = {}+    oneof_fields = defaultdict(list)++    if isinstance(protobuf_model, Descriptor):+        model_name = protobuf_model.name+        protobuf_fields = protobuf_model.fields+    elif isinstance(protobuf_model, GeneratedProtocolMessageType):+        model_name = protobuf_model.DESCRIPTOR.name+        protobuf_fields = protobuf_model.DESCRIPTOR.fields++    if model_name.endswith('Proto') and model_name in PROTO_TO_PYDANTIC_MODELS:+        return PROTO_TO_PYDANTIC_MODELS[model_name]++    for f in protobuf_fields:+        field_type = PROTOBUF_TO_PYTHON_TYPE[f.type]+        default_value = f.default_value++        if f.containing_oneof:+            # Proto Field type: oneof+            # NOTE: oneof fields are handled as a post-processing step+            oneof_fields[f.containing_oneof.name].append(f.name)++        if field_type is Enum:+            # Proto Field Type: enum+            enum_dict = {}+            for enum_field in f.enum_type.values:+                enum_dict[enum_field.name] = enum_field.number+            field_type = Enum(f.enum_type.name, enum_dict)++        if f.message_type:++            if f.message_type.name == 'Struct':+                # Proto Field Type: google.protobuf.Struct+                field_type = Dict+                default_value = {}++            elif f.message_type.name == 'Timestamp':+                # Proto Field Type: google.protobuf.Timestamp+                field_type = datetime+                default_value = datetime.now()++            elif f.message_type.name.endswith('Proto'):

I would just have an else here. I know it is true that now every Proto type ends with Proto, but is it really needed to introduce this assumption? At least we can add a static method in types describing if that is a Proto message, so that the logic is not spread in the codebase

deepankarm

comment created time in 7 hours

Pull request review commentjina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

-from typing import Callable, Dict, Any, Optional, List, Union+from typing import Dict, Any, Optional, List, Union++from enum import Enum+from datetime import datetime+from collections import defaultdict  from pydantic import Field, BaseModel, create_model+from google.protobuf.descriptor import Descriptor, FieldDescriptor+from google.protobuf.pyext.cpp_message import GeneratedProtocolMessageType  from jina.enums import DataInputType from jina.types.document import Document from jina.parsers import set_client_cli_parser-from jina.proto.jina_pb2 import DocumentProto, QueryLangProto+from jina.proto.jina_pb2 import (+    DenseNdArrayProto,+    NdArrayProto,+    SparseNdArrayProto,+    NamedScoreProto,+    DocumentProto,+    RouteProto,+    EnvelopeProto,+    StatusProto,+    MessageProto,+    RequestProto,+    QueryLangProto,+)+++DEFAULT_REQUEST_SIZE = set_client_cli_parser().parse_args([]).request_size+PROTO_TO_PYDANTIC_MODELS = {}+PROTOBUF_TO_PYTHON_TYPE = {+    FieldDescriptor.TYPE_INT32: int,+    FieldDescriptor.TYPE_INT64: int,+    FieldDescriptor.TYPE_UINT32: int,+    FieldDescriptor.TYPE_UINT64: int,+    FieldDescriptor.TYPE_SINT32: int,+    FieldDescriptor.TYPE_SINT64: int,+    FieldDescriptor.TYPE_BOOL: bool,+    FieldDescriptor.TYPE_FLOAT: float,+    FieldDescriptor.TYPE_DOUBLE: float,+    FieldDescriptor.TYPE_FIXED32: float,+    FieldDescriptor.TYPE_FIXED64: float,+    FieldDescriptor.TYPE_SFIXED32: float,+    FieldDescriptor.TYPE_SFIXED64: float,+    FieldDescriptor.TYPE_BYTES: bytes,+    FieldDescriptor.TYPE_STRING: str,+    FieldDescriptor.TYPE_ENUM: Enum,+    FieldDescriptor.TYPE_MESSAGE: None,+}+++def protobuf_to_pydantic_model(+    protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+) -> BaseModel:+    """+    Converts Protobuf messages to Pydantic model for jsonschema creation/validattion++    ..note:: Model gets assigned in the global dict :data:PROTO_TO_PYDANTIC_MODELS++    :param protobuf_model: *Proto message from proto file+    :type protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+    :return: Pydantic model+    :rtype: BaseModel+    """++    all_fields = {}+    oneof_fields = defaultdict(list)++    if isinstance(protobuf_model, Descriptor):+        model_name = protobuf_model.name+        protobuf_fields = protobuf_model.fields+    elif isinstance(protobuf_model, GeneratedProtocolMessageType):+        model_name = protobuf_model.DESCRIPTOR.name+        protobuf_fields = protobuf_model.DESCRIPTOR.fields++    if model_name.endswith('Proto') and model_name in PROTO_TO_PYDANTIC_MODELS:+        return PROTO_TO_PYDANTIC_MODELS[model_name]

Is this just an optimization?

Is it worth it the fact of introducing a global variable?

deepankarm

comment created time in 7 hours

Pull request review commentjina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

-from typing import Callable, Dict, Any, Optional, List, Union+from typing import Dict, Any, Optional, List, Union++from enum import Enum+from datetime import datetime+from collections import defaultdict  from pydantic import Field, BaseModel, create_model+from google.protobuf.descriptor import Descriptor, FieldDescriptor+from google.protobuf.pyext.cpp_message import GeneratedProtocolMessageType  from jina.enums import DataInputType from jina.types.document import Document from jina.parsers import set_client_cli_parser-from jina.proto.jina_pb2 import DocumentProto, QueryLangProto+from jina.proto.jina_pb2 import (+    DenseNdArrayProto,+    NdArrayProto,+    SparseNdArrayProto,+    NamedScoreProto,+    DocumentProto,+    RouteProto,+    EnvelopeProto,+    StatusProto,+    MessageProto,+    RequestProto,+    QueryLangProto,+)+++DEFAULT_REQUEST_SIZE = set_client_cli_parser().parse_args([]).request_size+PROTO_TO_PYDANTIC_MODELS = {}+PROTOBUF_TO_PYTHON_TYPE = {+    FieldDescriptor.TYPE_INT32: int,+    FieldDescriptor.TYPE_INT64: int,+    FieldDescriptor.TYPE_UINT32: int,+    FieldDescriptor.TYPE_UINT64: int,+    FieldDescriptor.TYPE_SINT32: int,+    FieldDescriptor.TYPE_SINT64: int,+    FieldDescriptor.TYPE_BOOL: bool,+    FieldDescriptor.TYPE_FLOAT: float,+    FieldDescriptor.TYPE_DOUBLE: float,+    FieldDescriptor.TYPE_FIXED32: float,+    FieldDescriptor.TYPE_FIXED64: float,+    FieldDescriptor.TYPE_SFIXED32: float,+    FieldDescriptor.TYPE_SFIXED64: float,+    FieldDescriptor.TYPE_BYTES: bytes,+    FieldDescriptor.TYPE_STRING: str,+    FieldDescriptor.TYPE_ENUM: Enum,+    FieldDescriptor.TYPE_MESSAGE: None,+}+++def protobuf_to_pydantic_model(+    protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+) -> BaseModel:+    """+    Converts Protobuf messages to Pydantic model for jsonschema creation/validattion++    ..note:: Model gets assigned in the global dict :data:PROTO_TO_PYDANTIC_MODELS++    :param protobuf_model: *Proto message from proto file+    :type protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+    :return: Pydantic model+    :rtype: BaseModel+    """++    all_fields = {}+    oneof_fields = defaultdict(list)++    if isinstance(protobuf_model, Descriptor):+        model_name = protobuf_model.name+        protobuf_fields = protobuf_model.fields+    elif isinstance(protobuf_model, GeneratedProtocolMessageType):+        model_name = protobuf_model.DESCRIPTOR.name+        protobuf_fields = protobuf_model.DESCRIPTOR.fields++    if model_name.endswith('Proto') and model_name in PROTO_TO_PYDANTIC_MODELS:+        return PROTO_TO_PYDANTIC_MODELS[model_name]++    for f in protobuf_fields:+        field_type = PROTOBUF_TO_PYTHON_TYPE[f.type]+        default_value = f.default_value++        if f.containing_oneof:+            # Proto Field type: oneof+            # NOTE: oneof fields are handled as a post-processing step+            oneof_fields[f.containing_oneof.name].append(f.name)++        if field_type is Enum:+            # Proto Field Type: enum+            enum_dict = {}+            for enum_field in f.enum_type.values:+                enum_dict[enum_field.name] = enum_field.number+            field_type = Enum(f.enum_type.name, enum_dict)++        if f.message_type:++            if f.message_type.name == 'Struct':+                # Proto Field Type: google.protobuf.Struct+                field_type = Dict+                default_value = {}++            elif f.message_type.name == 'Timestamp':+                # Proto Field Type: google.protobuf.Timestamp+                field_type = datetime+                default_value = datetime.now()

Isn't it better to put an stupid default value? Like this we can see if the timestamp is real or not? Normally timestamp is needed to measure some stuff about the presence of an object in a Pod, like this we try the "false positives" of seeing a meaningful timestamp

deepankarm

comment created time in 7 hours

Pull request review commentjina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

-from typing import Callable, Dict, Any, Optional, List, Union+from typing import Dict, Any, Optional, List, Union++from enum import Enum+from datetime import datetime+from collections import defaultdict  from pydantic import Field, BaseModel, create_model+from google.protobuf.descriptor import Descriptor, FieldDescriptor+from google.protobuf.pyext.cpp_message import GeneratedProtocolMessageType  from jina.enums import DataInputType from jina.types.document import Document from jina.parsers import set_client_cli_parser-from jina.proto.jina_pb2 import DocumentProto, QueryLangProto+from jina.proto.jina_pb2 import (+    DenseNdArrayProto,+    NdArrayProto,+    SparseNdArrayProto,+    NamedScoreProto,+    DocumentProto,+    RouteProto,+    EnvelopeProto,+    StatusProto,+    MessageProto,+    RequestProto,+    QueryLangProto,+)+++DEFAULT_REQUEST_SIZE = set_client_cli_parser().parse_args([]).request_size+PROTO_TO_PYDANTIC_MODELS = {}+PROTOBUF_TO_PYTHON_TYPE = {+    FieldDescriptor.TYPE_INT32: int,+    FieldDescriptor.TYPE_INT64: int,+    FieldDescriptor.TYPE_UINT32: int,+    FieldDescriptor.TYPE_UINT64: int,+    FieldDescriptor.TYPE_SINT32: int,+    FieldDescriptor.TYPE_SINT64: int,+    FieldDescriptor.TYPE_BOOL: bool,+    FieldDescriptor.TYPE_FLOAT: float,+    FieldDescriptor.TYPE_DOUBLE: float,+    FieldDescriptor.TYPE_FIXED32: float,+    FieldDescriptor.TYPE_FIXED64: float,+    FieldDescriptor.TYPE_SFIXED32: float,+    FieldDescriptor.TYPE_SFIXED64: float,+    FieldDescriptor.TYPE_BYTES: bytes,+    FieldDescriptor.TYPE_STRING: str,+    FieldDescriptor.TYPE_ENUM: Enum,+    FieldDescriptor.TYPE_MESSAGE: None,+}+++def protobuf_to_pydantic_model(

is it possible to break this down in smaller functions?

deepankarm

comment created time in 7 hours

pull request commentjina-ai/jina

@deepankarm really cool. Let's merge your pr first.

florian-hoenicke

comment created time in 7 hours

Pull request review commentjina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

-from typing import Callable, Dict, Any, Optional, List, Union+from typing import Dict, Any, Optional, List, Union++from enum import Enum+from datetime import datetime+from collections import defaultdict  from pydantic import Field, BaseModel, create_model+from google.protobuf.descriptor import Descriptor, FieldDescriptor+from google.protobuf.pyext.cpp_message import GeneratedProtocolMessageType  from jina.enums import DataInputType from jina.types.document import Document from jina.parsers import set_client_cli_parser-from jina.proto.jina_pb2 import DocumentProto, QueryLangProto+from jina.proto.jina_pb2 import (+    DenseNdArrayProto,+    NdArrayProto,+    SparseNdArrayProto,+    NamedScoreProto,+    DocumentProto,+    RouteProto,+    EnvelopeProto,+    StatusProto,+    MessageProto,+    RequestProto,+    QueryLangProto,+)+++DEFAULT_REQUEST_SIZE = set_client_cli_parser().parse_args([]).request_size+PROTO_TO_PYDANTIC_MODELS = {}+PROTOBUF_TO_PYTHON_TYPE = {+    FieldDescriptor.TYPE_INT32: int,+    FieldDescriptor.TYPE_INT64: int,+    FieldDescriptor.TYPE_UINT32: int,+    FieldDescriptor.TYPE_UINT64: int,+    FieldDescriptor.TYPE_SINT32: int,+    FieldDescriptor.TYPE_SINT64: int,+    FieldDescriptor.TYPE_BOOL: bool,+    FieldDescriptor.TYPE_FLOAT: float,+    FieldDescriptor.TYPE_DOUBLE: float,+    FieldDescriptor.TYPE_FIXED32: float,+    FieldDescriptor.TYPE_FIXED64: float,+    FieldDescriptor.TYPE_SFIXED32: float,+    FieldDescriptor.TYPE_SFIXED64: float,+    FieldDescriptor.TYPE_BYTES: bytes,+    FieldDescriptor.TYPE_STRING: str,+    FieldDescriptor.TYPE_ENUM: Enum,+    FieldDescriptor.TYPE_MESSAGE: None,+}+++def protobuf_to_pydantic_model(+    protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+) -> BaseModel:+    """+    Converts Protobuf messages to Pydantic model for jsonschema creation/validattion++    ..note:: Model gets assigned in the global dict :data:PROTO_TO_PYDANTIC_MODELS++    :param protobuf_model: *Proto message from proto file+    :type protobuf_model: Union[Descriptor, GeneratedProtocolMessageType]+    :return: Pydantic model+    :rtype: BaseModel+    """++    all_fields = {}+    oneof_fields = defaultdict(list)++    if isinstance(protobuf_model, Descriptor):+        model_name = protobuf_model.name+        protobuf_fields = protobuf_model.fields+    elif isinstance(protobuf_model, GeneratedProtocolMessageType):+        model_name = protobuf_model.DESCRIPTOR.name+        protobuf_fields = protobuf_model.DESCRIPTOR.fields++    if model_name.endswith('Proto') and model_name in PROTO_TO_PYDANTIC_MODELS:

this assumption is needed? isnt enough to remove this endswith? I think it adds a big assumption for the future

deepankarm

comment created time in 7 hours

pull request commentjina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

I like the conversion from protobuf to pydantic. It is more flexible in contrast to just adding the oneof-fields manually. Good work! @deepankarm

deepankarm

comment created time in 7 hours

pull request commentjina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

Codecov Report

Merging #2121 (2a5ae29) into master (c258e4a) will decrease coverage by 34.81%. The diff coverage is 1.51%.

@@             Coverage Diff             @@
##           master    #2121       +/-   ##
===========================================
- Coverage   88.34%   53.53%   -34.82%     
===========================================
  Files         211      189       -22     
  Lines       11104    10556      -548     
===========================================
- Hits         9810     5651     -4159     
- Misses       1294     4905     +3611

Flag	Coverage Δ
daemon	`?`
jina	`53.53% <1.51%> (-35.17%)`	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
jina/peapods/runtimes/asyncio/rest/models.py	`0.00% <0.00%> (-100.00%)`	:arrow_down:
jina/peapods/runtimes/asyncio/rest/app.py	`12.23% <16.66%> (-65.47%)`	:arrow_down:
jina/schemas/pod.py	`0.00% <0.00%> (-100.00%)`	:arrow_down:
jina/parsers/ping.py	`0.00% <0.00%> (-100.00%)`	:arrow_down:
jina/schemas/flow.py	`0.00% <0.00%> (-100.00%)`	:arrow_down:
jina/schemas/meta.py	`0.00% <0.00%> (-100.00%)`	:arrow_down:
jina/docker/helper.py	`0.00% <0.00%> (-100.00%)`	:arrow_down:
jina/schemas/driver.py	`0.00% <0.00%> (-100.00%)`	:arrow_down:
jina/parsers/hub/new.py	`0.00% <0.00%> (-100.00%)`	:arrow_down:
jina/schemas/request.py	`0.00% <0.00%> (-100.00%)`	:arrow_down:
... and 157 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 7dd876d...2a5ae29. Read the comment docs.

deepankarm

comment created time in 7 hours

push eventjina-ai/jina

Deepankar Mahapatro

commit sha 4f2ed0d0f85f930213b8cf1b3da7f0581d93293c

docs: fix return type

Deepankar Mahapatro

commit sha 2a5ae29df8edbfa16dc17c58d4f65bd37b116081

docs: fix docstrings

push time in 7 hours

pull request commentjina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

Thanks for your contribution :heart: :broken_heart: Unfortunately, this PR has one ore more bad commit messages, it can not be merged. To fix this problem, please refer to:

Note, other CI tests will not start until the commit messages get fixed.

This message will be deleted automatically when the commit messages get fixed.

deepankarm

comment created time in 7 hours

pull request commentjina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

Latency summary

Current PR yields:

😶 index QPS at 1217, delta to last 3 avg.: -1%
😶 query QPS at 21, delta to last 3 avg.: -1%

Breakdown

Version	Index QPS	Query QPS
current	1217	21
`1.0.7`	1240	21
`1.0.6`	1235	21

Backed by latency-tracking. Further commits will update this comment.

deepankarm

comment created time in 7 hours

push eventjina-ai/jina

Deepankar Mahapatro

commit sha 19c219d223d50f335b9318b4f92a84e65c2ca27e

doc: fix return type

Deepankar Mahapatro

commit sha 85c3ef6d8e2b62399a93c492a3567612ecad3c2b

doc: fix docstrings

push time in 7 hours

pull request commentjina-ai/jina

@florian-hoenicke I have fixed the issues with the response and the pydantic models in https://github.com/jina-ai/jina/pull/2121. We can focus only on tests in this PR.

florian-hoenicke

comment created time in 7 hours

PR opened jina-ai/jina

feat(schema): generate pydantic based jsonschema for any jina proto

+217 -37

1 commit in jina-ai/jina/tree/feat-proto-to-pydantic-schema jina-ai/jina/tree/master

0 comment

3 changed files

pr created time in 8 hours

create barnchjina-ai/jina

branch : feat-proto-to-pydantic-schema

created branch time in 8 hours

PR opened jina-ai/dashboard

Enhancement/209 command bar theme palette

Change background property of Button and ButtonGroup style to use props.theme.palette.primary instead of hard-coded hex color Closes #209

+19 -19

4 commit in rjgallego/dashboard/tree/enhancement/209_command_bar_theme_palette jina-ai/dashboard/tree/master

0 comment

1 changed file

pr created time in 11 hours

Pull request review commentjina-ai/jina

Improve Formatter

 def format(self, record):         :param record: A LogRecord object         :returns: Formatted LogRecord with level-colour MAPPING to add corresponding colour.         """-        cr = copy(record)+        cr = deepcopy(record)

Hi, thank you for the question. I suggested using deepcopy mostly because the record is a LogRecord object. Making a copy of the object should avoid most of the accidental overwrite of the original copy. However, I was a little worried on effect of codes from line 88-89 on the original object. I think even though it seems to be sufficient to just use copy for now, it might still be a better practice to make a completely independent copy of the LogRecord object into cr so that the integrity of the original copy would be preserved as much as possible. This might become more relevant if more operations are added with the cr variable.

chunyuema

comment created time in 11 hours

delete branch jina-ai/jina-hub

delete branch : chore-faissindexer-0.0.16-core-1-0-7

delete time in 14 hours

PR closed jina-ai/jina-hub

chore: testing/building FaissIndexer (0.0.16) on new jina core: 1.0.7 component/indexer size/XS type/manifest

Due to the release of jina core v1.0.7, this draft PR is created in order to trigger an automatic build & push of the module

+1 -0

1 commit in jina-ai/jina-hub/tree/chore-faissindexer-0.0.16-core-1-0-7 jina-ai/jina-hub/tree/master

1 comment

1 changed file

pr closed time in 14 hours

delete branch jina-ai/jina-hub

delete branch : chore-sptagindexer-0.0.13-core-1-0-7

delete time in 14 hours

pull request commentjina-ai/jina-hub

chore: testing/building FaissIndexer (0.0.16) on new jina core: 1.0.7

Automatic build successful. Image has been built and deployed.

jina-bot

comment created time in 14 hours

PR closed jina-ai/jina-hub

chore: testing/building SptagIndexer (0.0.13) on new jina core: 1.0.7 component/indexer size/XS type/manifest

Due to the release of jina core v1.0.7, this draft PR is created in order to trigger an automatic build & push of the module

+1 -0

1 commit in jina-ai/jina-hub/tree/chore-sptagindexer-0.0.13-core-1-0-7 jina-ai/jina-hub/tree/master

1 comment

1 changed file

pr closed time in 14 hours

pull request commentjina-ai/jina-hub

chore: testing/building SptagIndexer (0.0.13) on new jina core: 1.0.7

Automatic build successful. Image has been built and deployed.

jina-bot

comment created time in 14 hours

delete branch jina-ai/jina-hub

delete branch : chore-redisdbindexer-0.0.10-core-1-0-7

delete time in 14 hours