profile
viewpoint
shonenada shonenada @bearyinnovative Shenzhen, Guangdong, China

shonenada/Constance 4

Constance. A simple operating system.

shonenada/cGeDBIT 2

General purpose Database Indexing Technology in C++

shonenada/cwfg 1

chat room with websocket on flask + gevent

bearyinnovative/stalls 0

Applets for BearyChat

quickstarts/std-cms 0

CMS for SZU.

shonenada/acme.sh 0

A pure Unix shell script implementing ACME client protocol

shonenada/actix-web 0

Actix web is a small, pragmatic, and extremely fast rust web framework.

shonenada/amethyst 0

Data-oriented and data-driven game engine written in Rust

pull request commentredis/redis

Improve dbid check while selectDb

@oranagra I agree it's best for 6.2. Also please note I remember bumping into situations where the error text must be parsed for lack of distinct error codes where they could be useful. So while I don't think there's a problem with this specific change, we should be generally mindful to that.

sundb

comment created time in 42 minutes

starteddoawoo/teex

started time in an hour

issue closedredis/redis

Stack allocated RedisModuleString

I notice that there are some locations where the robj pass to a module callback as a RedisModuleString might be stack-allocated. This prevents the module writer from using RedisModule_RetainString on the given RedisModuleString, if the module writer will try to retain the given RedisModuleString he will crash on the assert.

While I believe its not a big production issue (the module writer will identify it on the testing stage probably). It might be inconvenient for the module writer because there is no way to know on the development stage if the given RedisModuleString is stack-allocated or not. In addition, there might be some rare code paths that are not tested and retain a stack-allocated RedisModuleString, those code paths might crash on production (totally the module writer fault of not testing his code but still it's confusing).

I can see 3 options to handle it:

  1. Document each function that receives a RedisModuleString whether or not it's ok to retain the given it.
  2. Introduce a new struct that will present the stack-allocated RedisModuleString. Notice that this option is good only if a function always returns either stack-allocated or heap-allocated RedisModuleString but not both. Unfortunately, there are functions that might return both in certain situations.
  3. Introduce a new function that gets RedisModuleString and return a copy of it. If the RedisModuleString is heap-allocated then it just increases the ref count and returns the same object. If the given RedisModuleString is stack-allocated then it creates a copy on the heap and returns it.

In my opinion, option 3 is the best. I believe that introducing such function and encourage module writers to use it instead of RedisModule_RetainString will allow, eventually, to deprecate RedisModule_RetainString.

Would like to hear your thoughts and opinions.

closed time in an hour

MeirShpilraien

issue commentredis/redis

Stack allocated RedisModuleString

Fixed by #7577.

MeirShpilraien

comment created time in an hour

issue closedredis/redis

Allow modules to expose their own api

Hey,

Today RedisModule allows low level operation on redis data type such as set string and push/pop from lists. As modules are able to create new datatypes, I find it helpful if modules could also expose their own api to allow other modules to use those new datatypes. I thought maybe its possible to add the 'moduleRegisterApi' function to the RedisModuleAPI, this way modules will be able to use it to register their api functions and other modules will be able to use 'RedisModule_GetApi' to get api function of other modules.

Thoughts?

closed time in an hour

MeirShpilraien

issue commentredis/redis

Allow modules to expose their own api

Closing as this was already implemented by 27f6e9bb9b (and a few others).

MeirShpilraien

comment created time in an hour

pull request commentredis/redis

Reset average ttl when empty databases

@ShooterIT what did you mean by the comment above about role change? It that just to state that if there's no full sync the information is kept (becomes stale)? I.E. Not affected by this PR. Or did you mean that on full sync it is reset (what this PR does)?

ShooterIT

comment created time in 2 hours

pull request commentpsf/requests

updated `get_encoding_from_headers` to return utf-8 if the content type is set to application/json

Awesome, thanks for the fast responses here and for merging!

jjmaldonis

comment created time in 2 hours

push eventpsf/requests

jason

commit sha 5855dd711f0ab9c9c4782574b8814b1e4c7f98cc

updated `get_encoding_from_headers` to return utf-8 if the content type is set to application/json, following RFC 4627. fixes #5667

view details

Ian Stapleton Cordasco

commit sha 589c4547338b592b1fb77c65663d8aa6fbb7e38b

Merge pull request #5673 from jjmaldonis/master updated `get_encoding_from_headers` to return utf-8 if the content type is set to application/json

view details

push time in 2 hours

PR merged psf/requests

updated `get_encoding_from_headers` to return utf-8 if the content type is set to application/json

This PR fixes #5667 by adding a small bit of code to get_encoding_from_headers that returns utf-8 if the content type is set to application/json and charset is not set. This follows RFC 4627 (ctrl+f for shall be) which reads: "JSON text SHALL be encoded in Unicode. The default encoding is UTF-8."

+4 -0

0 comment

1 changed file

jjmaldonis

pr closed time in 2 hours

issue closedpsf/requests

r.text and r.json() return different results *in some cases*

This might be an interesting one... I found that r.text and r.json() can return different results in some specific cases. I don't understand why the difference between the two cases is changing the result of r.text.

Maybe r.text should always default to using utf-8 as the decoding if application/json is set as the response's content type, following https://www.ietf.org/rfc/rfc4627.txt (ctrl+f for JSON text SHALL be encoded in Unicode.).

I'm using the latest version of requests.

Expected Result

I would expect r.text and r.json() to return ~ the same thing. More specifically, I would expect json.loads(r.text) and r.json() to return the same thing, but the issue seems to be with r.text's decoding specifically.

Actual Result

In the following code, I am making a sample request and replacing the request's response with custom content so we have full control over it. The custom content is utf-8 encoded. In the next version of this code, you'll see the name change when it shouldn't.

import requests
import json

r = requests.get("https://api.covidtracking.com/v1/us/current.json")
r._content = b'{"name":"rd\xce\xba"}'  # This is utf-8
r.headers = {
    "Content-Type": "application/json",
}
print(r.json())
print(json.loads(r.text))

The above code prints:

{'name': 'rdκ'}
{'name': 'rdκ'}

which is fantastic.

Replacing the request's content with b'{"name":"rd\xce\xba","uuid":"1234"}', which simply adds a uuid field to the JSON, and running the code again prints:

{'name': 'rdκ', 'uuid': '1234'}
{'name': 'rdκ', 'uuid': '1234'}

The name is different even though it did not change at all! The existence of "uuid":"1234" in the response's contents somehow changes the decoding. I have no clue why.

Reproduction Steps

Run this code:

import requests
import json

r = requests.get("https://api.covidtracking.com/v1/us/current.json")
r._content = b'{"name":"rd\xce\xba","uuid":"1234"}'
r.headers = {
    "Content-Type": "application/json",
}
print(r.json())
print(json.loads(r.text))

The issue should be fixed when the two print statements match... I think.

System Information

$ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.7"
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.4"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "1010103f",
    "version": "19.0.0"
  },
  "requests": {
    "version": "2.25.0"
  },
  "system_ssl": {
    "version": "1010104f"
  },
  "urllib3": {
    "version": "1.24.2"
  },
  "using_pyopenssl": true
}

closed time in 2 hours

jjmaldonis

issue commentpsf/requests

r.text and r.json() return different results *in some cases*

That's too bad about the issues with .text. If only it were possible to drop stuff like that. I totally agree with where you're coming from.

I just made a small PR based on what you said, it's #5673. I'm happy to make any changes you want.

jjmaldonis

comment created time in 2 hours

PR opened psf/requests

updated `get_encoding_from_headers` to return utf-8 if the content type is set to application/json

This PR fixes #5667 by adding a small bit of code to get_encoding_from_headers that returns utf-8 if the content type is set to application/json and charset is not set. This follows RFC 4627 (ctrl+f for shall be) which reads: "JSON text SHALL be encoded in Unicode. The default encoding is UTF-8."

+4 -0

0 comment

1 changed file

pr created time in 2 hours

Pull request review commentredis/redis

Iterate backwards on zdiff/zinter/zunion to optimize for zslInsert

 void zuiInitIterator(zsetopsrc *op) {             serverPanic("Unknown set encoding");         }     } else if (op->type == OBJ_ZSET) {+        /* Sorted sets are traversed in reverse order to optimize for+         * the insertion of the elements in a new list as in+         * ZDIFF/ZINTER/ZUNION */         iterzset *it = &op->iter.zset;         if (op->encoding == OBJ_ENCODING_ZIPLIST) {             it->zl.zl = op->subject->ptr;-            it->zl.eptr = ziplistIndex(it->zl.zl,0);+            it->zl.eptr = ziplistIndex(it->zl.zl,-2);             if (it->zl.eptr != NULL) {                 it->zl.sptr = ziplistNext(it->zl.zl,it->zl.eptr);

@felipou maybe we don't understand each other. i don't care if we go to -2, then -1, or go to -1 and then -2. but i think with the current code we iterate on the last item in the set twice. i.e. in our first iteration, eptr is -2, and sptr is -1. then we call zzlPrev, which will set sptr to -1, and etpr to -2. then call zzlPrev again, which will set sptr to -3 and eptr to -4.

or am i missing something?

felipou

comment created time in 4 hours

startedluncj/etcd-rs

started time in 4 hours

pull request commentredis/redis

Iterate backwards on zdiff/zinter/zunion to optimize for zslInsert

My initial thought was that this iterator (zuiInitIterator) should take a boolean input indicating if the iteration should be forward or backward, and support both, but if it's only gonna be used in one direction, i guess there's no need for that.

I started thinking something along those lines, I was going to implement a zuiInitReverseIterator and zuiPrev, but when I noticed that we could use the same direction in all cases, so I decided it was simpler to just always reverse the order.

Regarding the name, interesting observation about the "ui" prefix, but in that case we should have changed it to "uid" (union, intersection, diff), but IMHO that's lame. Note that the object it uses for the iteration is called "zsetopval" (i.e. "op" for "operation"). that's looks better IMHO, so if anything we can rename all of these to "op". However, i don't mind leaving them as is for now.

Ok, I'll leave them as is (good idea for the new name though).

I would like to ask that you conduct a simple benchmark on big objects to show that this change really improves performance (or at least doesn't hurt it). i.e. run ZUIONSTORE, ZINTERSTORE and ZDIFFSTORE (both algorithms) on large sets with the old and new code and measure the time (maybe using CONFIG RESETSTAT and INFO COMMANDSTATS). maybe do that once on skiplists, and once for ziplist (after setting zset-max-ziplist-entries to be very high).

IIUC in ZUNION, there's no benefit of doing backwards or forward, maybe we'll notice that forward if more cache friendly.

Ok, I'll try to run some benchmarks today, thanks for the tips!

felipou

comment created time in 4 hours

pull request commentredis/redis

Iterate backwards on zdiff/zinter/zunion to optimize for zslInsert

Heya @felipou and thanks for this. Would it possible to back up the "We prefer this order..." claim with a simple before/after benchmark?

@itamarhaber Yes, will do, I'll follow @oranagra suggestions!

felipou

comment created time in 4 hours

Pull request review commentredis/redis

Iterate backwards on zdiff/zinter/zunion to optimize for zslInsert

 void zuiInitIterator(zsetopsrc *op) {             serverPanic("Unknown set encoding");         }     } else if (op->type == OBJ_ZSET) {+        /* Sorted sets are traversed in reverse order to optimize for+         * the insertion of the elements in a new list as in+         * ZDIFF/ZINTER/ZUNION */         iterzset *it = &op->iter.zset;         if (op->encoding == OBJ_ENCODING_ZIPLIST) {             it->zl.zl = op->subject->ptr;-            it->zl.eptr = ziplistIndex(it->zl.zl,0);+            it->zl.eptr = ziplistIndex(it->zl.zl,-2);             if (it->zl.eptr != NULL) {                 it->zl.sptr = ziplistNext(it->zl.zl,it->zl.eptr);

Initially I implemented exactly as you said, and it worked, but then I realized that there is no need, just changing the initial index already does what we want. What matters for zzlPrev is that eptr points to the first element in the "pair", and sptr the second element. That's why it calls ziplistPrev twice, going back from eptr, the first call to ziplistPrev gets the previous sptr, and then gets the previous eptr.

In the initialization, we first get to eptr by getting the element just before the last one. The call to ziplistNext then gets the tail element, which is just after eptr. I can reverse the order like you said (get sptr at "-1", then eptr by using ziplistPrev), if you think it makes the code more readable, but it works the same.

felipou

comment created time in 4 hours

issue commentpsf/requests

r.text and r.json() return different results *in some cases*

So there's a semantic difference between .json() and .text. For one thing, I'd love for .text to not be a thing at all. Most of the time we guess wrong. If we dropped .text we wouldn't need chardet and we could finally end the ridiculous arguments about LGPL being involved in Requests.

Also, given the fact that .text does things while decoding bytes (like replace invalid sequences), I'd argue it should be avoided at all costs due to the likelihood of data corruption.

All this said, get_encoding_from_headers could probably smarter about application/json

jjmaldonis

comment created time in 4 hours

PR opened redis/redis

Reviewers
Adds 'use-memory' to GEORADIUS[BYMEMBER]

Partial resolution for #6860, item 7.

+2 -2

0 comment

1 changed file

pr created time in 5 hours

startedmylxsw/growing-up

started time in 6 hours

issue closedredis/redis

[BUG] CLANG compiler error on armv5 and armv7

Compiling redis on arm (32 bit) with clang results in the following errors (only stating two):

debug.c:963:36: error: no member named 'gregs' in 'mcontext_t' return (void*) uc->uc_mcontext.gregs[14]; /* Linux 32 */ ~~~~~~~~~~~~~~~ ^ debug.c:1137:41: error: no member named 'gregs' in 'mcontext_t' (unsigned long) uc->uc_mcontext.gregs[11], ~~~~~~~~~~~~~~~ ^ debug.c:1138:41: error: no member named 'gregs' in 'mcontext_t' (unsigned long) uc->uc_mcontext.gregs[8],

To reproduce

Clone the repo and compile on ARM using CLANG. Can only be reproduced on ARM and not on Intel using the same compiler suite. Using GCC on Arm works.

This can be reproduced with CLANG v11 running on Arch and CLANG v7.0.1 running on Debian.

Expected behavior

Compile w/o errors.

Additional information

Let me know if you need more information.

closed time in 7 hours

chrisAtRedis

issue commentredis/redis

[BUG] CLANG compiler error on armv5 and armv7

looks like this is already solved by #8095

chrisAtRedis

comment created time in 7 hours

Pull request review commentredis/redis

Adds pub/sub channel patterns to ACL

 int ACLCheckCommandPerm(client *c, int *keyidxptr) {     return ACL_OK; } +int ACLCheckPubsubChannelPerm(sds channel, list *allowed, int literal) {

Done.

itamarhaber

comment created time in 8 hours

Pull request review commentredis/redis

Adds pub/sub channel patterns to ACL

 acllog-max-len 128 # # requirepass foobared +# New users are initialized with restrictive permissions by default, via the+# equivalent of this ACL rule 'off -@all'. Starting with Redis 6.2, it is+# possible to manage access via ACL for Pub/Sub channels as well. To ensure+# backwards compatibility, new users are granted the 'allchannels' permission+# by default.+#+# Future compatibility note: it is very likely that in a future version of Redis+# the directive's default of 'allchannels' will be changed to 'resetchannels' in+# order to provide better out-of-the-box Pub/Sub security. Therefore, it is+# recommended that you explicitly define Pub/Sub permissions for all user rather+# then rely on the implicit defaults.+acl-pubsub-default allchannels

Done.

itamarhaber

comment created time in 8 hours

PR opened redis/redis

Reset average ttl when empty databases

If master changes into replica role, it still keep old average ttl.

+3 -0

0 comment

1 changed file

pr created time in 8 hours

issue commentredis/redis

Key length (>128) vs redis performance with keys command under lua

Hey @yossigo, I already seen PR for 5.4 https://github.com/redis/redis/pull/8078, code around this string hash is improved, but in sense that hash shift value is extracted to a define:

// from 5.4
/*
** Lua will use at most ~(2^LUAI_HASHLIMIT) bytes from a long string to
** compute its hash
*/
#if !defined(LUAI_HASHLIMIT)
#define LUAI_HASHLIMIT		5
#endif
...
unsigned int luaS_hashlongstr (TString *ts) {
  lua_assert(ts->tt == LUA_VLNGSTR);
  if (ts->extra == 0) {  /* no hash? */
    size_t len = ts->u.lnglen;
    size_t step = (len >> LUAI_HASHLIMIT) + 1;
    ts->hash = luaS_hash(getstr(ts), len, ts->hash, step);
    ts->extra = 1;  /* now it has its hash */
  }
  return ts->hash;
}

It's a step better, since after merging that to main branch, and including in my project as submodule, i can pass "-DLUAI_HASHLIMIt=8" parameter to CFLAGS to set it up. But i still need to have submodule and i still need to compile redis-server and create own deb redis-server package. This improvenemnt is removing necessary a patch file for lstring.c file. My preffered way would be to have this as as a redis.conf option, then i could take of the shelf redis-server package from ubuntu and just change redis.conf 1 line and would be good to go (which i already have my custom redis.conf anyway). Also we already established that making this change improves lua string hash collisions when having a lot of keys, and improves speed, which could be potentially desired for other users as well. Only downside is that it would require some ingerention redis/deps/lua source code.

kcudnik

comment created time in 9 hours

pull request commentredis/redis

Adds pub/sub channel patterns to ACL

also, please update the top comment to reflect the final version (with all the decisions and other changes in this PR), to be used as commit comment.

itamarhaber

comment created time in 9 hours

pull request commentredis/redis

Adds pub/sub channel patterns to ACL

@itamarhaber i noticed there are two checkboxes at the top not checked

itamarhaber

comment created time in 9 hours

more