profile
viewpoint
Salvatore Sanfilippo antirez Redis Labs Campobello di Licata, Sicily, Italy http://invece.org Computer programmer based in Sicily, Italy. I mostly write OSS software. Born 1977. Not a puritan.

antirez/disque 7581

Disque is a distributed message broker

antirez/kilo 4391

A text editor in less than 1000 LOC with syntax highlight and search.

antirez/linenoise 2380

A small self-contained alternative to readline and libedit

antirez/neural-redis 2162

Neural networks module for Redis

antirez/dump1090 1476

Dump1090 is a simple Mode S decoder for RTLSDR devices

antirez/lamernews 1339

Lamer News -- an HN style social news site written in Ruby/Sinatra/Redis/JQuery

antirez/hping 712

hping network tool

antirez/load81 462

SDL based Lua programming environment for kids similar to Codea

antirez/disque-module 382

Disque ported as Redis module

antirez/lua-cmsgpack 274

A self contained Lua MessagePack C implementation.

push eventantirez/redis

antirez

commit sha 20eeddfb8a8840aff8ac9e54a69650890d9c1e64

Signal key as modified when expired on-access. This fixes WATCH and client side caching with keys expiring because of a synchronous access and not because of background expiring.

view details

antirez

commit sha df45fed050d6822dc86796b0089aaf1b56c7830e

Merge branch 'unstable' of github.com:/antirez/redis into unstable

view details

push time in 3 days

push eventantirez/redis-doc

antirez

commit sha 5bde61698262d98f84c488cd1ad115f981f1dc46

Client side caching, update the information to match the new implementation.

view details

push time in 3 days

pull request commentantirez/redis

Fixes segfault on calling trackingGetTotalKeys

Thank you @itamarhaber!

itamarhaber

comment created time in 3 days

push eventantirez/redis

Itamar Haber

commit sha 8a44b2cc7ee15fedbacceced5d4a64d1b2a917c5

Fixes segfault on calling trackingGetTotalKeys ... with CSC disabled

view details

Salvatore Sanfilippo

commit sha ddb80bb3d483617787e57d29426a3e151e138534

Merge pull request #6890 from itamarhaber/patch-trackingGetTotalKeys Fixes segfault on calling trackingGetTotalKeys

view details

push time in 3 days

PR merged antirez/redis

Fixes segfault on calling trackingGetTotalKeys

... with CSC disabled

Reproducible by calling INFO.

+1 -0

0 comment

1 changed file

itamarhaber

pr closed time in 3 days

push eventantirez/redis

antirez

commit sha 92357b2d61bff014e3198bb39886d6b4e26163d1

Tracking: first conversion from hashing to key names.

view details

antirez

commit sha d933d6f2a42b9f0add28df212b5cfbc7756e3cf2

Tracking: rename INFO field with total items.

view details

antirez

commit sha 1ea66724300f1cd7c97b842a7c9190e72ea91976

Rax.c: populate data field after random walk.

view details

antirez

commit sha 85e4777d5c4411b499a845565729910bb09d64ac

Tracking: minor change of names and new INFO field.

view details

antirez

commit sha f53cc00c09a4e7c612b3781021246cbbeb533d7b

Tracking: always reply with an array of keys.

view details

antirez

commit sha dfe126f3e92c9770bef1915b6e64add2c41edcfa

Tracking: BCAST: parsing of the options + skeleton.

view details

antirez

commit sha 3f7ba86255b9d6acd73dd39cc8f05d3d3f8741a9

Tracking: BCAST: registration in the prefix table.

view details

antirez

commit sha 71f3f3f1afe4fbb6f8634970258b5dec2d389c68

Tracking: BCAST: broadcasting of keys in prefixes implemented.

view details

antirez

commit sha 40194a2a6809520b5f01da4a7b41afe2a2441f64

Tracking: BCAST: basic feature now works.

view details

antirez

commit sha 6922ccc0b98156e787b3d2f35daf0299e7844250

Tracking: fix sending messages bug + tracking off bug.

view details

antirez

commit sha f6e32a832f4aaa92721f4ea1eadc1d3897ba32c2

Tracking: fix behavior when switchinig from normal to BCAST.

view details

antirez

commit sha 47177c9edc9d6f738f1aacb33bd4e1d6c2c5a697

Tracking: fix operators precedence error in bcast check.

view details

antirez

commit sha 8ea7a3ee686a8bddf0b07585922917adcfda91dc

Tracking: first set of tests for the feature.

view details

antirez

commit sha 090bc0c1a37cfba092102527524ee2a3023d0481

Merge branch 'csc2' into unstable

view details

push time in 3 days

push eventantirez/redis

antirez

commit sha f6e32a832f4aaa92721f4ea1eadc1d3897ba32c2

Tracking: fix behavior when switchinig from normal to BCAST.

view details

antirez

commit sha 47177c9edc9d6f738f1aacb33bd4e1d6c2c5a697

Tracking: fix operators precedence error in bcast check.

view details

antirez

commit sha 8ea7a3ee686a8bddf0b07585922917adcfda91dc

Tracking: first set of tests for the feature.

view details

push time in 3 days

pull request commentantirez/redis

add no-slowlog option to RM_CreateCommand

+1

oranagra

comment created time in 4 days

push eventantirez/redis

Oran Agra

commit sha 46216b0e838b54099a0ac364b33b2dfb7f1de8c7

add no-slowlog option to RM_CreateCommand

view details

Salvatore Sanfilippo

commit sha c21c23bbba0989e84cb145163ea9ecd76f2db8fe

Merge pull request #6863 from oranagra/module_commands_no_slowlog add no-slowlog option to RM_CreateCommand

view details

push time in 4 days

push eventantirez/redis

antirez

commit sha 6922ccc0b98156e787b3d2f35daf0299e7844250

Tracking: fix sending messages bug + tracking off bug.

view details

push time in 4 days

push eventantirez/redis

antirez

commit sha 40194a2a6809520b5f01da4a7b41afe2a2441f64

Tracking: BCAST: basic feature now works.

view details

push time in 5 days

push eventantirez/sds

antirez

commit sha fb463145c9c245636feb28b5aac0fc897e16f67e

Make SDS_NOINIT extern instead of redefining it.

view details

push time in 5 days

push eventantirez/redis

Khem Raj

commit sha 3c610b4e8d8d4b09254c5e1a435ca25b82710e38

Mark extern definition of SDS_NOINIT in sds.h This helps in avoiding multiple definition of this variable, its also defined globally in sds.c Signed-off-by: Khem Raj <raj.khem@gmail.com>

view details

push time in 5 days

push eventantirez/redis

antirez

commit sha 577fc4388b4701d78244ec37d4dd8a5e7abe35ca

ACL LOG: data structures and initial functions.

view details

antirez

commit sha d9b153c9f66877a659ad2c87d178490cbd377749

ACL LOG: implement ACL LOG subcommadn skeleton.

view details

antirez

commit sha f1974d5d67a4b5489cfc4225da7bfe13ba10b563

ACL LOG: actually emit entries.

view details

antirez

commit sha e271a61103b8b57371e3283eddff5ef1b7b09716

ACL LOG: group similar entries in a given time delta.

view details

antirez

commit sha 943008ebac4ef23d30a2a32dd2900e25794811d5

ACL LOG: implement LOG RESET.

view details

antirez

commit sha 82790e510f30094efecfc865ffe4c7464c50825a

ACL LOG: also log ACL errors in the scripting/MULTI ctx.

view details

antirez

commit sha 9f6e84f6beabfc51e24580353ade795552cef8e2

ACL LOG: implement a few basic tests.

view details

antirez

commit sha 7379c78a9b584ca5f394bae7ae7139ee92fa88bb

ACL LOG: log failed auth attempts.

view details

antirez

commit sha ea1e1b12c94c3e17707136d0d859a610260a3bea

ACL LOG: test for AUTH reason.

view details

antirez

commit sha 51c1a9f8fbc12a9276489178242e498bb6ccbdba

ACL LOG: make max log entries configurable.

view details

Oran Agra

commit sha c82ccf0670465ae9ee45aa711b8f27b224aa365e

memoryGetKeys helper function so that ACL can limit access to keys for MEMORY command

view details

Oran Agra

commit sha 488e194787ee1e0bece37ca2d555e767e5e43372

config.c verbose error replies for CONFIG SET, like config file parsing We noticed that the error replies for the generic mechanism for enums are very verbose for config file parsing, but not for config set command. instead of replicating this code, i did a small refactoring to share code between CONFIG SET and config file parsing. and also renamed the enum group functions to be consistent with the naming of other types.

view details

Yossi Gottlieb

commit sha 736309660f7531c059690184c17798010b8155e8

TLS: Some redis.conf clarifications.

view details

Guy Benoish

commit sha fae306b3745556458a9f4900cdcbbce4affbcb71

Fix small bugs related to replica and monitor ambiguity 1. server.repl_no_slaves_since can be set when a MONITOR client disconnects 2. c->repl_ack_time can be set by a newline from a MONITOR client 3. Improved comments

view details

Guy Benoish

commit sha 5a6cfbf4ca737be71178624ef721e6dd053fb645

Some refactroing using getClientType instead of CLIENT_SLAVE

view details

Oran Agra

commit sha a55e5847067a6c0a68371756b290d77f8f91d25d

DEBUG HELP - add PROTOCOL

view details

Oran Agra

commit sha df096bc96bad620a89797bf94298f53a227ff76b

add SAVE subcommand to ACL HELP and top comment

view details

Oran Agra

commit sha f42ce57d0f7eee6705fb0a81714cabe97a4f2c0a

stopAppendOnly resets aof_rewrite_scheduled althouh in theory, users can do BGREWRITEAOF even if aof is disabled, i suppose it is more common that the scheduled flag is set by either startAppendOnly, of a failed initial AOFRW fork (AOF_WAIT_REWRITE)

view details

Oran Agra

commit sha ba289244185fd717c4ebc52ae6ab02d9b7bcdd25

move restartAOFAfterSYNC from replicaofCommand to replicationUnsetMaster replicationUnsetMaster can be called from other places, not just replicaofCOmmand, and all of these need to restart AOF

view details

Oran Agra

commit sha 22e45d46fe8aeb1fa385fdca5db80eac6776a290

freeClientAsync don't lock mutex if there's just one thread

view details

push time in 5 days

push eventantirez/redis

antirez

commit sha f53cc00c09a4e7c612b3781021246cbbeb533d7b

Tracking: always reply with an array of keys.

view details

antirez

commit sha dfe126f3e92c9770bef1915b6e64add2c41edcfa

Tracking: BCAST: parsing of the options + skeleton.

view details

antirez

commit sha 3f7ba86255b9d6acd73dd39cc8f05d3d3f8741a9

Tracking: BCAST: registration in the prefix table.

view details

antirez

commit sha 71f3f3f1afe4fbb6f8634970258b5dec2d389c68

Tracking: BCAST: broadcasting of keys in prefixes implemented.

view details

push time in 5 days

pull request commentantirez/redis

Mark extern definition of SDS_NOINIT in sds.h

Thank you @kraj, back porting on Redis 5 and 4 as well.

kraj

comment created time in 5 days

push eventantirez/redis

Khem Raj

commit sha af02478ba02da8320d80661ccbf91a5b125b03ab

Mark extern definition of SDS_NOINIT in sds.h This helps in avoiding multiple definition of this variable, its also defined globally in sds.c Signed-off-by: Khem Raj <raj.khem@gmail.com>

view details

Salvatore Sanfilippo

commit sha 8aa0b19d833195e01f90192bb9bfec6124343230

Merge pull request #6691 from kraj/fno-common Mark extern definition of SDS_NOINIT in sds.h

view details

push time in 5 days

PR merged antirez/redis

Mark extern definition of SDS_NOINIT in sds.h

This helps avoiding multiple definition of this variable, its also defined globally in sds.c

Signed-off-by: Khem Raj raj.khem@gmail.com

+1 -1

1 comment

1 changed file

kraj

pr closed time in 5 days

push eventantirez/redis

lifubang

commit sha dc8f947d7c8a26881c3a5789a2dbf8d3a5ef946d

correct help info for --user and --pass Signed-off-by: lifubang <lifubang@acmcoder.com>

view details

Salvatore Sanfilippo

commit sha f54bb2a3309482e80b4ee3e7c8bf5bb724baa70e

Merge pull request #6882 from lifubang/userpass correct help info for --user and --pass

view details

push time in 5 days

pull request commentantirez/redis

correct help info for --user and --pass

Thank you @lifubang

lifubang

comment created time in 5 days

PR merged antirez/redis

correct help info for --user and --pass

fix #6880 in redis-cli -h:

-user --> --user -pass --> --pass

In #6880 The last one is not a bug.

Signed-off-by: lifubang lifubang@acmcoder.com

+2 -2

0 comment

1 changed file

lifubang

pr closed time in 5 days

issue closedantirez/redis

redis-6.0-rc1 acl related issues

Took 6.0 RC1 ACL feature for a drive today on MacOS 10.15.1. Pretty impressive. Listing here some of the things that I noticed. Apologize if these have been reported before by others.

  1. The redis-server and redis-cli start up message says Redis 5.9.101

  2. redis-cli -user does not work as advertised redis-rules:> redis-cli --help 2>&1 | grep user -user <username> Used to send ACL style 'AUTH username pass'. Needs -a. -pass <password> Alias of -a for consistency with the new --user option. redis-rules:> redis-cli -user u1 Unrecognized option or bad number of args for: '-user' redis-rules:> redis-cli -user u1 -a abc Unrecognized option or bad number of args for: '-user'

--user seem to work

  1. redis-cli -pass does not work as advertised redis-rules:> redis-cli -pass abc Unrecognized option or bad number of args for: '-pass'

--pass seem to work

  1. Command gets executed even when the AUTH fails redis-rules:> redis-cli --user non-exist --pass abc ping
    Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. Warning: AUTH failed PONG

closed time in 5 days

vkasar

issue commentantirez/redis

Client Side Caching and TTL interaction

@eliblight what you suggest is an anti-pattern modifying just single commands. Redis would be a jungle if we went that way :-) I think you have to do some effort to detach yourself from the problem you are solving for your customers and think to everybody in the community, I understand it is very tempting to solve the problem you have at hand in the most efficient way, but in that way we end with a very good hammer that is efficient only with certain kinds of nails. Later we can say too: GET is so important that we want to specialize it in a similar way to what you suggest, but the first step is to have an orthogonal feature that works for every command. Also note that in practice, there is basically no difference: CACHING is in a pipeline and is not going to take more TCP packets in the average case. The same for the reply. So it only looks more expansive, but is actually only more flexible.

Btw I'm interested in getting feedbacks regarding the "csc2" branch.

eliblight

comment created time in 8 days

push eventantirez/redis

Seunghoon Woo

commit sha 16b2d07f0a9b58027611dab7f97788d37cb5ab84

[FIX] revisit CVE-2015-8080 vulnerability

view details

push time in 8 days

push eventantirez/redis

Seunghoon Woo

commit sha ef764dde1cca2f25d00686673d1bc89448819571

[FIX] revisit CVE-2015-8080 vulnerability

view details

Salvatore Sanfilippo

commit sha 256ec6c52f5bd41437ea703801f67426af370918

Merge pull request #6875 from WOOSEUNGHOON/cve20158080_fix [FIX] revisit CVE-2015-8080 vulnerability

view details

push time in 8 days

pull request commentantirez/redis

[FIX] revisit CVE-2015-8080 vulnerability

Thanks! I'm backporting the fix in the Redis 5 branch.

WOOSEUNGHOON

comment created time in 8 days

PR merged antirez/redis

[FIX] revisit CVE-2015-8080 vulnerability

Hi,

We discovered that the CVE-2015-8080 vulnerability revisited in the latest version of Redis (5.0.7).

  • Initial issue number: #2855
  • Initial fix commit: 3a47c8cfb85af1b69cccf30eaaa690e4a23ab20a (Dec. 2015)

The vulnerability is from the Lua source code that you already patched in Dec. 2015. However, as a result of the Lua update in May 2018 (commit: 1eb08bcd4634ae42ec45e8284923ac048beaa4c3), the vulnerability patch was removed during the update process.

As a result, we successfully reproduce the vulnerability in the latest stable version of Redis using simple PoC provided in issue #2855.

Please reflect it after confirmation. Thank you.

+6 -4

0 comment

1 changed file

WOOSEUNGHOON

pr closed time in 8 days

push eventantirez/try.redis

antirez

commit sha 8bf94197004ee6a0f89b454a88d614652bb2665e

Fix try.redis link.

view details

push time in 9 days

push eventantirez/try.redis

antirez

commit sha 803a2d57e76a68dc05dbf42aae970262f550a218

Update the footer to reflect current setups.

view details

push time in 9 days

issue commentantirez/redis

Client Side Caching and TTL interaction

Caching uses a pattern already used in Redis cluster ASKING command and in other places too, so people should have no issues with it and it is general, does not require any additional modification to commands, no round trip more since you can always pipeline it, and so forth.

On Sun, Feb 9, 2020, 07:11 Yossi Gottlieb notifications@github.com wrote:

I think the auto-clearing property of the CACHING command is a bit confusing, as it introduces a new paradigm which is not used elsewhere. Then there's the matter of efficiency when batch fetching keys.

The GET flags proposed by @eliblight https://github.com/eliblight solve this and make things more explicit, but it focuses only on strings and ignores other data types.

What if we leverage the existing MULTI state for that purpose? i.e. the opt-in (or opt-out) would be an optional MULTI flag which will affect everything in it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/antirez/redis/issues/6833?email_source=notifications&email_token=AAAQAYHQ6HPW2D2ICB7DZWLRB6NBLA5CNFSM4KQLUKFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELGDOJI#issuecomment-583808805, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAQAYAJRFTRVGTFHURDQGLRB6NBLANCNFSM4KQLUKFA .

eliblight

comment created time in 9 days

issue commentantirez/redis

Client Side Caching and TTL interaction

Hello @eliblight there was some misunderstanding here, this is how it works, when you set Caching to yes it is auto cleared after the next call or transaction executed. You need yes and no in order to also support OPTOUT version.

eliblight

comment created time in 9 days

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

There is an implementation using keys instead of hashes in the crc2 branch here.

eliblight

comment created time in 10 days

push eventantirez/rax

antirez

commit sha 23f31ca89c669841a4dc9dda73fe09aa6f014f28

Rax.c: populate data field after random walk.

view details

push time in 10 days

create barnchantirez/redis

branch : csc2

created branch time in 10 days

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

@eliblight one way in order to alleviate the problem you describe, for which there is basically no solution, is the following: detect keys that have a very short client-side ability to live cached, and don't cache them for some time. This is simple but requires client smartness, however here is the client that drives the whole protocol, so it makes totally sense that it is able to avoid caching the wrong values based on simple statistic take in the client side.

eliblight

comment created time in 10 days

issue commentantirez/redis

Client Side Caching and TTL interaction

@itamarhaber note that in RESP3 we also have attributes, so later when a client is in tracking mode, if we set a special option like "GETTTL", we can have the attribute for the key ttl returned as we fetch things. But this is an incremental development that we can add if needed, no need to over-design this now.

eliblight

comment created time in 10 days

issue commentantirez/redis

Client Side Caching and TTL interaction

@yossigo, I agree with everything you said but not letting the client handle TTL is not any better than to do. Applications depends on proper TTL and CSC should not break that.

CSC does not break it, clients that want to cache keys with a TTL can tag keys with the TTL and expire them upon access. Apps where this is not critical will just set a fixed max TTL for cached value, and wait for invalidation messages from Redis.

eliblight

comment created time in 10 days

issue commentantirez/redis

Client Side Caching and TTL interaction

@itamarhaber this is just a so common case worth optimizing, for all the rest there is MULTI/TTL/CMD/EXEC.

eliblight

comment created time in 10 days

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

On the server side, one thing that still bothers me is we can cap the overhead that is derived from the number of tracked keys, but not from the number of clients.

I think we are already able to have a maximum cap on the number of objects, even if I tend to believe we don't need that. Right now, we'll initially cap the number of keys we track, but as we touch the keys, to have an estimate of the number of average client IDs per key is not hard to calculate. Based on that, we can later (if really needed) cap the number of objects, either by expiring more keys depending on the amount of data they hold, or alternatively, by expiring IDs from the keys, and sending invalidations only to a selected amount of clients.

The reason I believe this is not going to be useful, is that actually CSC is a distributed system that involves clients and whose efficiency has a lot to do with access pattern in the whole data space: users will have to tune the tradeoff of memory they want to spend in the server side VS the latency they want to obtain based on their exact work load. For instance in heavy write workloads the expires will keep coming and the server will have the opposite problem: not being able to track anything and flushing continuously the key->IDs map and sending invalidation messages.

eliblight

comment created time in 11 days

issue commentantirez/redis

Client Side Caching and TTL interaction

Please make sure to read this: https://github.com/antirez/redis/issues/6867

eliblight

comment created time in 11 days

issue openedantirez/redis

Client side caching: new developments pending

This is a list of things I'm changing in CSC:

  1. Try to rewrite the code to track keys instead of hash slots as per suggestion of @eliblight.
  2. Implement the CLIENT TRACKING on OPTIN to only track keys that are fetched after a "CACHING yes" command. Maybe also support OPTOUT and "CACHING no" before the next command. Note that this schema is about race conditions: we want to prepare the next command to be executed+tracked and we don't want to inform later the server that we are tracking a key, otherwise is a nightmare.
  3. Make sure keys are invalidated in expireIfNeeded(), @yossigo pinged about that and we need to verity.
  4. Implement the BCAST mode plus the PREFIX support as explained in https://www.youtube.com/watch?v=TW9uFIJ9xkc
  5. Implement a WITHTTL option for the GET command returning the milliseconds TTL of the fetched key as suggested by @eliblight.
  6. Change invalidation messages in order to have an array of elements and not a single element for invalidation messages, this is very important in the case of BCAST, but may also be used to improve the normal tracking feature, accumulating data for clients and sending them only at the end.

Document the following things:

  1. Race condition between invalidation messages and data because of the two channels design:
Data: -> GET foo
Notifications: <- Invalidate foo (somebody else touched it)
Data: <- "bar" (foo value)

We need to inform client authors that they need to create the entry for "foo" before sending the command. If an invalidation message is received in the meantime, we need to discard the cached "foo" key (that would be set to "value will arrive" sentinel initially), and when the actual value arrives we don't do anything with it, since the entry is missing in the cache table of the client.

  1. Inform the client authors that invalidations because of TTLs may be delayed, but will surely be received, so they may implement either always a maximum TTL depending on the application demands, or implement proper TTL handling in the client, fetching the TTL with the value itself when doing caching.

created time in 11 days

issue commentantirez/redis

Client Side Caching and TTL interaction

So... I think I don't agree with this issue for different reasons. To start I'll reply to @yossigo: if we don't send the invalidation on expireIfNeeded(), this is just a bug, probably I thought that it would be sent as a side effect of deleting a key when we perform signalModifiedKey(), are you sure this is not happening? I'll try, but anyway that would just be a bug.

Then the reasons why I don't agree with @eliblight about all this:

  1. In general CSC violates certain consistency guarantees, because the tracking channel may have a different delay compared to the data channel. This delta can be reduced however, by continuously pinging the Pub/Sub channel (or RESP3 connection) in order to detect if the invalidations channel is broken, and in that case, flushing all the local client side data (that is a very bold action). However there is no general solution for that, it's a tradeoff.
  2. The deletion notification that we get with CSC for keys with a TTL, can be considered a "best effort" mechanism, we can document that clients that want more precise TTL expiration should also get the TTL. Moreover if we don't do it already, we should make sure to send an invalidation message also when the TTL is changed from a key.
  3. We need to warn the client authors that in general when implementing this protocol, they should set a reasonable TTL for the data anyway, this is already outlined in the draft CSC documentation that we have online.
  4. The expire mechanism is more likely to evolve in the future and become more precise. Work was already in progress but didn't reach maturity for Redis 6, but in the future we'll very probably end with precise TTLs.

About your point "GET_TRACKED_KEY", not sure if you are aware, but in the CSC specification before Redis 6 gets GA, there is already such a mode where you specify with a command (after enabling tracking), only the keys you want to track, so this is not needed from the POV of fine grained tracking.

This is what I read in the current design document I've here:

CLIENT TRACKING on OPTIN
CACHING (yes/no?)
GET foo

Basically you send CACHING yes if you want to cache the next value, it flags the client and will track the keys returned in the next script / transaction / command. Note that this is much better from the POV of race conditions compared to saying later "I'm caching xyz".

I'm quite skeptical about doing any change like that. Of this whole proposal, I agree only on a single point, that is that a GET option to return the PTTL of a key could be nice to have in general and even more now.

eliblight

comment created time in 11 days

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

@yossigo well, we may not be so far from the final design, because the suggestion of @eliblight can be just taken as, TLDR, replace indexing by hash with indexing by key name, still taking a capped collection (via eviction as we had to do, anyway, even with caching slots), and send the key instead of the ID. Here the main tradeoffs are: less efficiency (hashing + indexing the slot is faster thank looking up) and more memory usage (but well, we have a way to decide the number of keys), VS a big simplification of the concepts to expose to the user (not my main concern) and better scalability (if you have memory, you can let the client cache as many keys as you want without compromising the precision of the caching because of collisions. So far I think that Eran may have a good point here and that such sacrifices could be in the right spirit. Initially the reason I avoided doing this, is because I missed the fact (realized later and implemented) that I could evict stuff by sending non-real eviction messages to clients.

eliblight

comment created time in 11 days

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

As far as design... I see no difference (for the sake of discussion) between adding to the current redix key->val+invalidated clients or creating a duplicate of just key->invalidated clients.

I totally agree that for the sake of software design... keeping them decoupled might make a lot of sense... and have no strong opinion either way...

The big difference is that decoupling you need a way to limit number of keys stored in CSC side. But deleting a random one is not a problem AFAIK, every time we reach the level.

eliblight

comment created time in 11 days

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

However note that we have to resort to random key eviction to don't go over a maximum configured limit of keys that we store server side. But this should not hurt the feature much.

eliblight

comment created time in 11 days

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

I'm now intrigued by the @eliblight main proposal of trying to send the keys. I'll evaluate the impacts considering we can't implement stuff in a much different way (I don't want to mix in any way CSC with the main dictionary implementation for many reasons). Probably it could be a good idea indeed. Let me reason about that a bit.

eliblight

comment created time in 11 days

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

@antirez My comment might have been miscommunicated, I think BCAST is still useful, I think it solves a different problem in the same space. I believe when eran is proposing is more for replacing the the large tracking table.

This is turning into slack :D

Oh ok, I didn't understand your comment. So we are on the same page on BCAST being useful as low-memory, high-messages mode.

eliblight

comment created time in 11 days

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

And to compensate for the used memory, we can use a more efficient representation of the list of IDs.

eliblight

comment created time in 11 days

IssuesEvent

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

@eliblight we can't do what you suggested, but we can modify the current implementation to use just a radix tree (of key names) of radix trees (of client IDs).

eliblight

comment created time in 11 days

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

So, even if we switch to sending the whole key, that may be a reasonable path especially because of what I said above, that is Eran point about the fact that when we, anyway, reach collisions we have no longer much to gain, and the fact that I envisioned random eviction (and sending of the notifications) only later when I started implementing this design, that can be used in order to cap the number of keys that we remember, the BCAST mode would be still super useful, because it is a different mode where we tell Redis: don't remember anything, just send me all the notifications for all the keys with such prefixes.

eliblight

comment created time in 11 days

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

@madolson that's not too bad because clients may just hash the key when received and use the other mechanism if they want a single implementation, btw your comment makes sense indeed. The second part does not IMHO, BCAST is still needed, is a completely different system with other tradeoffs, see: https://www.youtube.com/watch?v=TW9uFIJ9xkc&t=915s

eliblight

comment created time in 11 days

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

@yossigo yep good question. I want to @eliblight to make sure to consider that any potential design will have to evaluated similarly. That is, I didn't touched the main keyspace for reasons (including decoupling and memory usage of the keys object when nothing like that is used), so memory usage must be evaluated, to start, imagining that we don't have an additional pointer in the key space. Otherwise it is not an apple to apple comparison at all. You can surely remodel your argument, saying that instead of having a linear array -> radix tree of client IDs, we could have a radix tree of keys -> radix tree of client IDs. When when there is a key explosion problem, we do what we do now, that is, to have a capped size of keys.

A good point of this design, is that we can store access informations soon or later, and when evicting keys from the CSC table, we can evict keys that are less requested. But it will use significantly more memory, we could evaluate it, and understand if anyway it's a better design. I'm starting to think that this design has some validity, but not for most of the reasons exposed above, like the hash function being part of the API or client complexity, but more for one of the reasons that Eran said about, anyway, having as a goal a limited amount of collisions.

Another thing that we can do in order to bring the memory back at the current level while STILL having the system that Eran told us, is to use a linear sorted array of IDs instead of the radix tree up to a given number of clients.

eliblight

comment created time in 11 days

issue commentantirez/redis

[Client Side Cashing] Communicate hashed keys, do we really need it?

Please make sure to read this thread with the following note: the BCAST mode communicates the whole key, since in that case is for free. I'll comment more in the next days, for now implementing BCAST inside Redis so that we have the full CSC design in place as of version 1.

eliblight

comment created time in 11 days

push eventantirez/redis

Guy Benoish

commit sha 92dc5e1fa41491e0ab0744a2bab55f837db89dc2

Diskless-load emptyDb-related fixes 1. Call emptyDb even in case of diskless-load: We want modules to get the same FLUSHDB event as disk-based replication. 2. Do not fire any module events when flushing the backups array. 3. Delete redundant call to signalFlushedDb (Called from emptyDb).

view details

Salvatore Sanfilippo

commit sha 9c00bdd86e8b5e8e863c056cf7c1eb69a00fdc51

Merge pull request #6822 from guybe7/diskless_load_module_hook_fix Diskless-load emptyDb-related fixes

view details

push time in 12 days

PR merged antirez/redis

Diskless-load emptyDb-related fixes
  1. Call emptyDb even in case of diskless-load: We want modules to get the same FLUSHDB event as disk-based replication.
  2. Do not fire any module events when flushing the backups array.
  3. Delete redundant call to signalFlushedDb (Called from emptyDb).
+44 -28

3 comments

3 changed files

guybe7

pr closed time in 12 days

pull request commentantirez/redis

Diskless-load emptyDb-related fixes

@guybe7 About "2", I get that, I mean right now in my source tree I can't see emptyDb() already calling signalFlushedDb(). But now I see that I mis-understood your comment "Delete redundant call to signalFlushedDb (Called from emptyDb).", You mean that it's already called from emptyDb() so you removed it from the replication code. Perfect, merging, thanks for clarifying.

guybe7

comment created time in 12 days

pull request commentantirez/redis

Diskless-load emptyDb-related fixes

Hi @guybe7, a few notes:

  1. Please could you document the new flag in the top comment?
  2. emptyDb() does not call signalFlushedDb() AFAIK.
guybe7

comment created time in 12 days

push eventantirez/redis

antirez

commit sha 3e9e27e98fa7ecdbb5a34676e51cda54de671d8a

ACL LOG: data structures and initial functions.

view details

antirez

commit sha e8d0057710d11b3361daf37b93e479947aebee5a

ACL LOG: implement ACL LOG subcommadn skeleton.

view details

antirez

commit sha 61dffd8669bdc16f26c67c0ec8955a0d8a627ef8

ACL LOG: actually emit entries.

view details

antirez

commit sha 6671032fafc82be04b4ca564790753b26f96780e

ACL LOG: group similar entries in a given time delta.

view details

antirez

commit sha 30a466ba38759e8deb5ed79c18bb3d3dec352ad7

ACL LOG: implement LOG RESET.

view details

antirez

commit sha 396161765b4f44f84fe428576a4272003669cec9

ACL LOG: also log ACL errors in the scripting/MULTI ctx.

view details

antirez

commit sha b189a2197489d20cf233d447f961e8a9c3b4009a

ACL LOG: implement a few basic tests.

view details

antirez

commit sha 0c1a4b557691df2196d41835aeef5c1f29914b82

ACL LOG: log failed auth attempts.

view details

antirez

commit sha 64a73e9293154482977cd530a2adc05f1fcc92f6

ACL LOG: test for AUTH reason.

view details

antirez

commit sha 90fae58b49cbc0bf0be76fe889952a81f4c3aed1

ACL LOG: make max log entries configurable.

view details

antirez

commit sha d5c6a833c8e7fa0f64c0a9a51f8a567690190448

Merge branch 'acl-log' into unstable

view details

antirez

commit sha 50d4326e3b65b4ab713a9ab8bd59f36a4b66b52f

Merge branch 'unstable' of github.com:/antirez/redis into unstable

view details

push time in 12 days

pull request commentantirez/redis

fix ssl args check for redis-cli

Merged, thank you @lifubang

lifubang

comment created time in 12 days

push eventantirez/redis

lifubang

commit sha 540b917a262c376f187dda51b4a7988aadef7ef1

fix ssl flag check for redis-cli Signed-off-by: lifubang <lifubang@acmcoder.com>

view details

Salvatore Sanfilippo

commit sha 1012514e0f10aa090eb39c870b82b10724d4d667

Merge pull request #6826 from lifubang/opensslcli fix ssl args check for redis-cli

view details

push time in 12 days

PR merged antirez/redis

fix ssl args check for redis-cli

redis-cli doesn't check ssl args, so if we don't pass a value to it, it will cause redis-cli aborted.

root@DESKTOP-UVUP1SF:/opt/redis# /opt/redis/src/redis-cli --tls  --cert /opt/myredis/redis.crt   --key /opt/myredis/redis.key   --cacert
zmalloc: Out of memory trying to allocate 18446744073709551608 bytes
Aborted (core dumped)

I think we should add lastarg check for these args and change ssl args's comments in redis-cli --help.

Signed-off-by: lifubang lifubang@acmcoder.com

+10 -9

2 comments

1 changed file

lifubang

pr closed time in 12 days

pull request commentantirez/redis

RM_Scan disable dict rehashing

@oranagra thanks. About the doc, it should explain the user why they can't play with the dictionary in the mean time. You should write the API will return all the keys that were consistently there from the start to the end of the iteration, but duplicates may be returned, and the more you play with the dict, the more duplicates you may get, so to get less duplicates it could be a good idea to avoid modifying the dictionary. Otherwise the reader will just wonder why and what happens.

oranagra

comment created time in 12 days

pull request commentantirez/redis

Add RM_CreateStringFromDouble

We can still go for 128 but apparently it is wrong, the maximum length of a double string should be something like ~325 chars or alike after a quick check. We may add this broken as it is in the rest of Redis and later fix everything when we will fix everything. Makes sense?

guybe7

comment created time in 12 days

pull request commentantirez/redis

Exclude "keymiss" notification from NOTIFY_ALL

Thanks @guybe7

guybe7

comment created time in 12 days

push eventantirez/redis

Guy Benoish

commit sha 2fda5f5c98ca0f0ce426c7dff3951804db62e0fe

Exclude "keymiss" notification from NOTIFY_ALL Because "keymiss" is "special" compared to the rest of the notifications (Trying not to break existing apps using the 'A' format for notifications) Also updated redis.conf and module.c docs

view details

Salvatore Sanfilippo

commit sha 5bfba06a3bddc7041987652f8d33d6cd42c9603b

Merge pull request #6821 from guybe7/key_miss_notify Exclude "keymiss" notification from NOTIFY_ALL

view details

push time in 12 days

PR merged antirez/redis

Exclude "keymiss" notification from NOTIFY_ALL

Because "keymiss" is "special" compared to the rest of the notifications (Trying not to break existing apps using the 'A' format for notifications)

+12 -7

0 comment

5 changed files

guybe7

pr closed time in 12 days

pull request commentantirez/redis

Add RM_CreateStringFromDouble

Hey @guybe7, it's ok but char buf[128]; is wrong, we have a define for that size.

guybe7

comment created time in 12 days

pull request commentantirez/redis

Optimize temporary memory allocations for getKeysFromCommand mechanism

Totally agree but 65536 is huge without really improving anything compared to, like, 64... because 99.99% of the times it's <= a small number.

oranagra

comment created time in 12 days

push eventantirez/redis

Oran Agra

commit sha 3795aaf42aadf46b415edaa801383cd81085eb91

update RM_SignalModifiedKey doc comment

view details

Salvatore Sanfilippo

commit sha 2e1dd00c8013d5f0b9c7067ede1b74be5ceb3c45

Merge pull request #6837 from oranagra/signal_modified_key_doc update RM_SignalModifiedKey doc comment

view details

push time in 12 days

pull request commentantirez/redis

RM_Scan disable dict rehashing

Thanks Oran, why we don't want this to happen directly at dictScan() API?

oranagra

comment created time in 12 days

pull request commentantirez/redis

Add handling of short read of module id in rdb

+1

oranagra

comment created time in 12 days

push eventantirez/redis

Oran Agra

commit sha fe7e8dc955ef6c5f3b843f87763f62f6eb84bd4b

Add handling of short read of module id in rdb

view details

Salvatore Sanfilippo

commit sha 730cacf67272356ca50fa34684a4c67e1a64ecba

Merge pull request #6840 from oranagra/short_read_moduleid Add handling of short read of module id in rdb

view details

push time in 12 days

pull request commentantirez/redis

TLS: Update documentation.

Thanks Yossi.

yossigo

comment created time in 12 days

push eventantirez/redis

Yossi Gottlieb

commit sha bb3d45a38683fc97c0b9b06ff7725fa1eca5d80c

TLS: Update documentation.

view details

Salvatore Sanfilippo

commit sha 4abba65ec730bdfd75fca8e2ea4f833b179538f6

Merge pull request #6841 from yossigo/tls-doc-update TLS: Update documentation.

view details

push time in 12 days

PR merged antirez/redis

TLS: Update documentation.

Hi @antirez here are some minor updates to the TLS related documentation. I'll also approach the redis.io documentation sooner than later.

+32 -31

0 comment

2 changed files

yossigo

pr closed time in 12 days

pull request commentantirez/redis

Add user/client name to MONITOR and SLOWLOG

Hi @guybe7, during the previous meeting on Slack we said we want to design this with more efforts before making breaking changes. Thanks.

guybe7

comment created time in 12 days

push eventantirez/redis

Oran Agra

commit sha ac2c96f5b1714b36f215fc9caec87e746a804041

A few non-data commands that should be allowed while loading or stale SELECT, and HELLO are commands that may be executed by the client as soon as it connects, there's no reason to block them, preventing the client from doing the rest of his sequence (which might just be INFO or CONFIG, etc). MONITOR, DEBUG, SLOWLOG, TIME, LASTSAVE are all non-data accessing commands, which there's no reason to block.

view details

Salvatore Sanfilippo

commit sha 33f613bf87d02229e893d6e288b3b84697351a13

Merge pull request #6843 from oranagra/command_flags A few non-data commands that should be allowed while loading or stale

view details

push time in 12 days

PR merged antirez/redis

A few non-data commands that should be allowed while loading or stale

SELECT, and HELLO are commands that may be executed by the client as soon as it connects, there's no reason to block them, preventing the client from doing the rest of his sequence (which might just be INFO or CONFIG, etc).

MONITOR, DEBUG, SLOWLOG, TIME, LASTSAVE are all non-data accessing commands, which there's no reason to block.

+8 -8

1 comment

1 changed file

oranagra

pr closed time in 12 days

pull request commentantirez/redis

A few non-data commands that should be allowed while loading or stale

Agreed.

oranagra

comment created time in 12 days

pull request commentantirez/redis

Memory leak when bind config is provided twice

Thanks

oranagra

comment created time in 12 days

push eventantirez/redis

Oran Agra

commit sha 7e1d5954e59ab99c572f2dccccf6843c9aad9127

Memory leak when bind config is provided twice

view details

Salvatore Sanfilippo

commit sha 4aafdb185ae4840a233338988728caa8cf5624c9

Merge pull request #6844 from oranagra/bind_config_leak Memory leak when bind config is provided twice

view details

push time in 12 days

pull request commentantirez/redis

fix maxmemory config warning

Thank you.

oranagra

comment created time in 12 days

push eventantirez/redis

Oran Agra

commit sha a17bddf2a14a1bd85a3606d2023fb9bb92004770

fix maxmemory config warning the warning condition was if usage > limit (saying it'll cause eviction or oom), but in fact the eviction and oom depends on used minus slave buffers. other than fixing the condition, i add info about the current usage and limit, which may be useful when looking at the log.

view details

Salvatore Sanfilippo

commit sha be520829b84a60338c3a672b9c14c67aab3076d9

Merge pull request #6845 from oranagra/maxmemory_warning fix maxmemory config warning

view details

push time in 12 days

PR merged antirez/redis

fix maxmemory config warning

the warning condition was if usage > limit (saying it'll cause eviction or oom), but in fact the eviction and oom depends on used minus slave buffers.

other than fixing the condition, i add info about the current usage and limit, which may be useful when looking at the log.

+3 -2

0 comment

1 changed file

oranagra

pr closed time in 12 days

push eventantirez/redis

Oran Agra

commit sha d454d5a4f516991c0f1e21a285c71f38a113e297

Fix client flags to be int64 in module.c currently there's no bug since the flags these functions handle are always lower than 32bit, but still better fix the type to prevent future bugs.

view details

Salvatore Sanfilippo

commit sha 13741fb99d803a72154d2ed9d7d0eb8f0ed864f1

Merge pull request #6846 from oranagra/module_client_flags Fix client flags to be int64 in module.c

view details

push time in 12 days

PR merged antirez/redis

Fix client flags to be int64 in module.c

currently there's no bug since the flags these functions handle are always lower than 32bit, but still better fix the type to prevent future bugs.

+3 -3

1 comment

1 changed file

oranagra

pr closed time in 12 days

pull request commentantirez/redis

Fix client flags to be int64 in module.c

Good, could hide bugs in a very subtle way in the future. Thanks.

oranagra

comment created time in 12 days

pull request commentantirez/redis

moduleRDBLoadError, add key name, and use panic rather than exit

Anyway we exit(1)ed... so. Thanks.

oranagra

comment created time in 12 days

push eventantirez/redis

Oran Agra

commit sha 85cc696f50064327e81a41952ffb2be830048701

moduleRDBLoadError, add key name, and use panic rather than exit using panic rather than exit means you get s stack trace of the code path that experianced the error, and possibly other info.

view details

Salvatore Sanfilippo

commit sha 08b218bfa5ac0b6cebd91e18af080cc105d26c10

Merge pull request #6847 from oranagra/module_read_err_panic moduleRDBLoadError, add key name, and use panic rather than exit

view details

push time in 12 days

PR merged antirez/redis

moduleRDBLoadError, add key name, and use panic rather than exit

using panic rather than exit means you get s stack trace of the code path that experianced the error, and possibly other info.

+5 -4

0 comment

1 changed file

oranagra

pr closed time in 12 days

pull request commentantirez/redis

reduce repeated calls to use_diskless_load

Thank you.

oranagra

comment created time in 12 days

push eventantirez/redis

Oran Agra

commit sha 485d5d4a18441feff626e61e31e480c3054fe877

reduce repeated calls to use_diskless_load this function possibly iterates on the module list

view details

Salvatore Sanfilippo

commit sha 5558c0e4cf85a1745c7d574fd6c7adadddb1adac

Merge pull request #6848 from oranagra/opt_use_diskless_load_calls reduce repeated calls to use_diskless_load

view details

push time in 12 days

PR merged antirez/redis

reduce repeated calls to use_diskless_load

this function possibly iterates on the module list

+3 -4

0 comment

1 changed file

oranagra

pr closed time in 12 days

push eventantirez/redis

Oran Agra

commit sha 86e302f5f302da89790f03000e9f76d322b5f971

freeClientAsync don't lock mutex if there's just one thread

view details

Salvatore Sanfilippo

commit sha 7cf53252eedd0251aeb66be8513a3a7f1fd4b30d

Merge pull request #6849 from oranagra/free_client_mutex freeClientAsync don't lock mutex if there's just one thread

view details

push time in 12 days

more