Let's add put_and_get function to mail API of Cachex.

How it should work

Function put_and_get should work similar to put, but return additional element – value:

iex > Cachex.put_and_get(:my_cache, "key", "value")
{:ok, true, "value"} 
iex >

Motivation

In some use cases we use flow like:

def do_something(key) do

{_, value} = Cachex.get(:my_cache, key)

  case value do
     nil -> 
          new_value = some_heavy_computation(key)
          Cachex.put(:my_cache, key, new_value)
          new_value # the important line
    _ -> value
  end

end

Now if you omit the important line, then you get nil on return from do_something. This change is ‘quality of life‘ change, that protect developers against some type of errors.

Impact to existing API

Nope

This is already covered by using get_and_update/4. Please see the examples in the docs: https://hexdocs.pm/cachex/Cachex.html#get_and_update/4

This is implementation of #258

Yes, if you use import/3 and export/2 - you can export and re-import your data after a restart.

Hello! Would be possible to republish the jar artifacts? These 3 links are returning 404 not found

https://oss.sonatype.org/content/repositories/releases/io/whitfin/siphash/2.0/siphash-2.0.jar https://repo1.maven.org/maven2/io/whitfin/siphash/2.0/siphash-2.0.jar https://repo.maven.apache.org/maven2/io/whitfin/siphash/2.0/siphash-2.0.jar

Thanks

Hi @volpatorafael! You should be using 2.0.0, not 2.0. Everything is working as expected :)

Fixes #11 and fixes #9.

Hopefully this makes it easier to build on Windows, where uniq doesn't exist - so this library would be more useful.

Depending on a C library complicates the compilation, since it requires that the Clang compiler be present.

xxHash has already been implemented in Rust in the twox-hash crate.

Using it would probably make it easy to publish pre-built binaries for macOS, Linux and Windows, which would improve the usability of runiq :)

error: failed to run custom build command for xxhash-sys v0.1.0

Caused by: process didn't exit successfully: C:\Users\Admin\AppData\Local\Temp\cargo-installfAw9KD\release\build\xxhash-sys-696d500ada48d2a1\build-script-build (exit code: 101) --- stdout running: "cmake" "C:\Users\Admin\.cargo\registry\src\github.com-1ecc6299db9ec823\xxhash-sys-0.1.0\xxHash/cmake_unofficial" "-G" "Visual Studio 16 2019" "-Thost=x64" "-Ax64" "-DCMAKE_INSTALL_PREFIX=C:\Users\Admin\AppData\Local\Temp\cargo-installfAw9KD\release\build\xxhash-sys-fa8f6d7a1b4e633c\out" "-DCMAKE_C_FLAGS= -DXXH_NAMESPACE=_rust_xxhash_sys -nologo -MD -Brepro" "-DCMAKE_C_FLAGS_RELEASE= -DXXH_NAMESPACE=_rust_xxhash_sys -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS= -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS_RELEASE= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS_RELEASE= -nologo -MD -Brepro" "-DCMAKE_BUILD_TYPE=Release"

--- stderr thread 'main' panicked at ' failed to execute command: The system cannot find the file specified. (os error 2) is cmake not installed?

build script failed, must exit now', C:\Users\Admin.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.44\src\lib.rs:885:5 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

Hopefully this makes it easier to build on Windows, where uniq doesn't exist - so this library would be more useful.

See comments in the related issue :) thank you though!

Would it be possible to add a -c flag to output a count of each unique line, like uniq has? A significant proportion of my usage of uniq is in the form of sort | uniq -c | sort -n, and being able to use runiq to replace that initial pair of commands would be really nice.

Hi!

Sorry, haven't had time to look at this in a while. In this case though, there's really no need to add this. You can do the following:

cat file.txt | runiq | wc -l

As discussed. Fixes #2.

Apologies if I'm not understanding some Rust conventions but it seems to me that that the docs are misleading about the types yielded when iterating over lines. E.g.

// do something with the line, which is &[u8]

whereas really (AIUI) it's Result<&[u8], Error>. The same is true of several of the comments (and thus in the official docs on crates.io).

• Using LocalCluster.start_nodes/3 with option :files in an umbrella project corrupted the ExUnit test management. The test file was re-compiled and re-executed for each subsequent umbrella app. A warning was emitted for each re-compile: warning: redefining module XXX (current version defined in memory)
• This change avoids code changes on the master node by moving the compilation to each slave node (assuming the compiler is loaded there).

Note that this PR changes implementation for #8. Will it work for you? Is there a better solution? Please advise.

@pmenhart ok after reviewing this I'm going to accept your change; I have no idea why v1.2 fails - it's something to do with the anonymous functions but I don't see much difference between the stdlib code for v1.2 and v1.3 here, so it feels like just a limitation of v1.2 somehow.

v1.2 still works with this lib, it's just that the anonymous functions do not. I don't think this is a major issue as there are ways around it and very few people are using v1.2 anyway, so I'm just going to merge as is.

Are you including your test files? That's the only thing I can imagine, and that seems quite odd.

I'll take another look at this today and push a patch; let me think about 1.2 for a while.

@pmenhart yeah it makes sense given that you can use it to exclude; I missed that initially but realised when merging. Thanks for the work! I'm going to push a new version soon with that PR included; sorry again for the delay.

In order to pass a different environment into the remote node when starting it, the user must change their local Application env, start the node, then change it back. For simple cases this works fine, but once tests are running in parallel, this can cause chaos.

This PR adds an :env_override option that allows the user to inject different environmental variables into the remote node. This is especially useful when each group of nodes in a test must start up on a different port or with different app options enabled.

Build passed: https://travis-ci.org/github/whitfin/local-cluster/builds/667949124

Regardless, I merged your changes inside https://github.com/whitfin/local-cluster/commit/523545b0bdc7cb077e8afb080d01ed35d4e53a8e - with a tweaking of the naming to environment and a slight touch of reformatting. Thanks for the contribution!

I see that application configs are passed to spawned nodes here https://github.com/whitfin/local-cluster/blob/f86e6f043c0d58a2ea7bb44f7185a54a3694b949/lib/local_cluster.ex#L70 And looks like it works, but sometimes in my umbrella project some applications are terminating because of missing configs. I have 2 suggestions

• some umbrella app somehow is getting started before this line is executed https://github.com/whitfin/local-cluster/blob/f86e6f043c0d58a2ea7bb44f7185a54a3694b949/lib/local_cluster.ex#L70

• some umbrella app is not loaded (and loading later with missing config) https://github.com/whitfin/local-cluster/blob/f86e6f043c0d58a2ea7bb44f7185a54a3694b949/lib/local_cluster.ex#L68

is it possible?

I think this is related to https://github.com/whitfin/local-cluster/issues/13 which is resolved by #15. It should be fixed in the next release (or current master). I'm going to close this, but feel free to re-open if you still have issues!

Hello!

Tried this library and got permanent :not_alive error on LocalCluster.start_nodes Also your library test returned this

00:02:37.767 [info]  Protocol 'inet_tcp': register/listen error: econnrefused                                                                                                                                                   
                                                                                                                                                                                                                                
** (MatchError) no match of right hand side value: {:error, {{:shutdown, {:failed_to_start_child, :net_kernel, {:EXIT, :nodistribution}}}, {:child, :undefined, :net_sup_dynamic, {:erl_distribution, :start_link, [[:"manager@1
27.0.0.1"], false]}, :permanent, 1000, :supervisor, [:erl_distribution]}}}                                                                                                                                                      
    test/test_helper.exs:2: (file)                                                                                                                                                                                              
    (elixir) lib/code.ex:813: Code.require_file/2                                                                                                                                                                               
    (elixir) lib/enum.ex:783: Enum."-each/2-lists^foreach/1-0-"/2    

I am not sure - may be this issue is trivial and self obvious for true erlang developers, but for noobs like me it was kind of pain.

Finally I got the reason - there must be epmd daemon running.

So may be reflect this requirements in documentation? It seems mandatory condition so looks logical for me to write couple of lines about it.

I added a small note about this in the README, thanks!

When starting the nodes the current implementation does not take in account applications deps, which results in application started out of order and configurations wrongly merged. Using Mix allow to start in the correct order with the very same environment that the developer expects.

So... I looked this over and I believe it can be resolved by #15, which has been merged. Want to try on latest master?

I'm going to close for now, but feel free to re-open if you think it should be!

• Using LocalCluster.start_nodes/3 with option :files in an umbrella project corrupted the ExUnit test management. The test file was re-compiled and re-executed for each subsequent umbrella app. A warning was emitted for each re-compile: warning: redefining module XXX (current version defined in memory)
• This change avoids code changes on the master node by moving the compilation to each slave node (assuming the compiler is loaded there).

Note that this PR changes implementation for #8. Will it work for you? Is there a better solution? Please advise.

On second look, there's actually no issue here - the warning is just because the string is being compiled repeatedly. This is not an issue though because you will already have loaded the same module anyway, so the warning is redundant. I don't think we need to "fix" this. The warning is annoying, but it shouldn't be harming anything.

Please review df9be34 mainly, other commits are unrelated to main issue.

The current implementation copies application config (Application.put_env/4) and starts loaded applications (Application.ensure_all_started/2) simultaneously. But it can start an application before copying its configuration if other application starts it.

So in this PR, I modified code to starts applications after all configuration merged.

I believe this is fixed with the merge of #15, so I'm going to close it for now. Please test with the latest master and verify that it is; if not let me know and we can re-open!

Prior to this, applications were started in a non-deterministic order, which could lead to misconfiguration or applications not booting. Now you can specify a root app that will be started prior to all the other applications. Due to the way that applications are loaded, specifying this single app should be enough to boot correctly.

I've merged this in the meantime, since the naming doesn't block the merge. I'm likely going to tweak the option to :applications rather than :app_names, but the rest looks good!

• Default behaviour: start_nodes/3 starts all applications that are loaded on the master node. There are situations where some apps are not desirable on slave nodes, or where their start order is causing problems.
• New option :app_names accepts an ordered list of applications to be started. Very useful for test scenarios where only a subset of apps should be copied over, or where the order of applications starting is critical.
• Note that each application is started via Application.ensure_all_started/2, i.e. with all dependencies.

The remote log level wasn't being set correctly due to the subtleties of how Elixir handles keyword lists.

[a: :b] gets interpreted as [{:a, :b}], but the args list needs [[{:a, :b}]], since the keyword list is just one of multiple possible arguments. This patch gets the log level set correctly.

This fixes an issue (specifically with propcheck) where the other nodes would fail to boot. Propcheck tries to access it's context dets file at boot and having multiple instances launch at the same time causes errors like:

06:42:50.228 [info]  Application propcheck exited:
   PropCheck.App.start(:normal, []) returned an error: shutdown: failed
to start child: PropCheck.CounterStrike
    ** (EXIT) an exception was raised:
        ** (MatchError) no match of right hand side value: {:error,{:not_a_dets_file, '_build/propcheck.ctex'}}
            (propcheck 1.2.0) lib/counterstrike.ex:53:PropCheck.CounterStrike.init/1
            (stdlib 3.9.2) gen_server.erl:374: :gen_server.init_it/2
            (stdlib 3.9.2) gen_server.erl:342: :gen_server.init_it/6
            (stdlib 3.9.2) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

The solution for this is to allow local cluster to skip loading certain dependencies when booting the other nodes.

Yes, I'm going to close this in favour of #15 - which I am intending to merge in some form once I get some clarification on another thread. Thanks for the PR though!

Seems fine to me, but the 1.2 build is curious - any ideas why?

Based on code alone, IMO, #15 is indeed the way to go seeing as you can accomplish what this PR does already, with extras. @scohen I believe your "root" app would simply be the first app in the multiple apps list in #15, thus making this PR redundant in the grand scheme of things.

However I don't really understand the multiple apps thing either; what use case would require starting multiple apps in a specific order where you wouldn't only spawn the root app, as @scohen says?

@keathley I'm not sure what you have included in that fork, but if it's only a few of these PRs and you've been using them in production, I'm happy to try get it merged back here and push a release.

I think I also prefer :include rather than :files, thoughts?

I would prefer :applications, to be in line with other places such a list is used.

Hi, yes it's active - sorta. I have little time for Elixir these days - and this project is low on a list of already low priorities, particularly given the global situation.

I'll take a look at this PR and the linked PR though, and see if we can't get them in soon :)

I'm not sure I understand the use case; why would your test environment have different options to the environment you're running your tests under? What's the use case that you can't just change them in your local env?

Hi, I've added this test in order to try to illustrate my thoughts on how the API was going to work. I can understand if there are reasons why it shouldn't work that way but for my use case and my level of understanding, it would be a really useful addition.

I'm afraid I would not be confident in trying change things so that this test passed and I'm not sure if that would be desirable anyway.

Please close this PR if this isn't a reasonably direction to go in.

Aimed at helping: #253

The 'ttl: 1' is not respected and so the entry doesn't expire so we don't get the "used" value back from the fallback on the fetch.

The assert on key10 works but because the ttl isn't set rather than because the ttl is long enough that it shouldn't have expired.

It is not expected that this works; the ttl option is not supported with fetch. See here.

Closes #242

FWIW, I have been running this fork in production for a few weeks now without any issues.

I have an application in which I store some externally managed configuration in Cachex. Without this configuration, the application is completely unusable, and should not accept any reqeuests.

I use a Preemptive Warmer to hydrate this cache, but since the warmer is executed asynchronously, my application finishes booting, and begins accepting requests before the warmer has finished executing.

Would you be open to a PR adding an option to use Cachex.Warmer, maybe something like block_on_startup: true, which would cause the execute callback to be executed during init?

Thanks for the amazing library!

I couldn't get it to happen, but I'll check again in case I missed anything.

As for the usage of transactions, we want to make sure the fallback function doesn't get called concurrently for the same key.

This is automatically handled for you; if two calls happen for the same key, the first one will do the fetch and the second call will just await the result of the first one. You shouldn't have to do it in a transaction; if you see behaviour differently to this, there's a bug somewhere.

Hi @sneako, I'm sorry it took so long to get around to this!

There are two things I want to discuss here; the first is that I believe this should be considered a bug, and as such your optional change should be our only behaviour. As it stands in the latest version, applications can go live without potentially vital data (as is your case). This is definitely an oversight. It also means that if your warmer is failing, your application might go live and simply never have cached data (since a crashing warmer would not stop your application starting). This also feels like a mistake.

I'm interested in your thoughts, seeing as you hit it naturally. I think we should perhaps just avoid adding an option, and just change the current behaviour to block. I can't think of a compelling argument against this; the only negative I can think of is that startup time might be longer, but you're trading that for safety so it's always worth it. I'm pretty convinced this is the route we should take, but I want to make sure I'm not missing anything before we do so. One thing I'm concerned about is people using the current version who do not update or use an option, and they don't realise this is happening (or has the potential to happen).

If we do decide to have an option, I don't like that this PR places it on the Warmer options itself. I'd much rather add an option to the warmer record passed to the cache startup, so you could start the same Warmer with/without blocking. It feels like a more logical place to put it, I think. I'm also not a huge fan of the option name, I'd prefer something simpler like :blocking, but that's kinda nitpick.

Either way, good report! Depending what you think, I'd like to get a solution to this merged and shipped in a minor version. If we decide to go without the option, it should be a minor change that we can get in quickly. I'll await your response for a while; but if you're busy there's a chance I'll grab this at some point to bundle into a release I'm planning shortly for compatibility purposes.

As per discussion in #195, minor enhancements to docs to clarify limitations of :ttl option.

I'm going to close this as I lifted it into the current state of https://github.com/whitfin/cachex/blob/master/docs/features/ttl-implementation.md - let me know if that looks good to you!

I'm not going to change the API docs themselves just yet; I need to figure out what I want first. I'm considering removing the option entirely in the next major version of Cachex (whenever that may be), since I think it causes confusion and isn't worth the slight performance gain you get from it. It would leave us with only two options; either a cache wide default, or expire/4. Much better, I think?

Just trying to improve a bit the lib's documentation for new users.

I've taken your changes from this up in some other changes I've made to the README, so I'm going to close this for now. Thanks for the contributions!

When using async tests with a shared Postgres connection, clearing a cache key in the test code does not clear the cache key in the server-side.

If you would inspect the cache in the test, it will be empty but when you inspect the key on the app side it isn't.

I would like to disable the cachex for the test, or clear the cache key in the setup block only as described the the Cachex.reset(:organisations) does not work.

This is only the case for Async tests.

I've read https://github.com/whitfin/cachex/issues/213 i think the same issue applies here, only the question was not relayed properly.

Thanks!

Yes, this is the same question as the other thread. I read it again and thought my final response seemed a little harsh (not quite the word I want to use), but then I read it once again and I believe my response is actually the same.

Logically, the only way this can be an issue is if you have two tests running at the same time, both operating on the same cache key. In this scenario, the flaw lies in your test code. The Cachex.reset/1 function already does what you are asking for; and yes it does work, because it's what I use in the library tests themselves (it's why this function exists). The root cause of your issue is that you have one test clearing your key, whilst the other test is putting it back.

I'm not sure if this issue is asking for changes, but there is no way to stop this happening than to either:

• Use a different key in each test
• Run your tests sequentially (non-async)

This isn't something that can be "fixed" on the side of Cachex, because the logic issue is inside your test setup. The same concept applies to your question about disabling the cache for your tests; I'm not the person starting the cache - it's in your code somewhere. In the place that is done, you can check for the Mix environment and disable the cache accordingly.

Let me know if this makes sense; I'm going to close this in the meantime!

Hello, I'm having some trouble understanding some guarantees of Cachex as a cache mechanism for databases (using Ecto with Postgres in my case).

For example, let's say I want to add a new value to the database and at the same time add it to the cache list, something like this:

{:ok, _} =  MyRepo.insert(value)

Cachex.get_and_update(:my_cache, "key", fn
  (nil) -> {:ignore, key}
  (list) -> {:commit, [value | list]}
end)

What I don't understand is if functions like get_and_update are guaranteed to succeed. If that is not guaranteed, it means that in the above example I can insert the value to the database, and then get_and_update fails for some reason making my cache outdated as it will never contain the value when I use get or fetch.

I guess I can make sure I don't have that problem by using an Ecto.transaction with Multi and adding the get_and_update as the last step of Multi. that way the database write would rollback if Cachex call failed.

Is that necessary or I can safely use something like the above example as long as my fn for the third parameter is not broken?

The best way to say this is that it's undefined. In the current codebase, these things are basically guaranteed - but you should react to the result that is returned and handle it accordingly. This ensures that you are protected in case Cachex ever changes the underlying implementation such that it's no longer guaranteed.

Doing what you said with transactions makes a lot of sense. Another pattern is to avoid manually updating the cache, and instead use a warmer or a fallback mechanism with a TTL. LMK if you need some examples!

I was reading issue 148 and I was wondering if/when this project might be updated to use the new child specifications (Elixir 1.5+)?

In issue 148 @whitfin mentions what I assume is to refactor closer to the Supervisor.Spec being removed from the Elixir library, i.e. Elixir 2.0+.

I recall a talk by José that Elixir would be a long time before that happens. This is good, as Supervisor.Spec and worker/3 will continue to work.

This is more of a question if that refactor is in the roadmap, or if this is a desired refactor for others to do?

... Forgot to add thanks for the cool project.

Hey everyone! Sorry it took forever (fairly distracted with current situations) but I've updated the signatures based on the advice from José above, and everything looks good. I updated the documentation too, so it should be more familiar. Please let me know if anything is encouraging a bad pattern; I haven't written much Elixir since the spec changes so I want to make sure I'm not portraying anything incorrectly.

I tested the changes with a few dependent packages from Hex and I'm pretty sure everything is fine in terms of backwards compatibility too, so should be good! I'm going to close this, but please do re-open if anything needs improvement. Sorry again for the delay!

I'm trying to upgrade from cachex 3.1.3 to 3.2.0 and some tests are failing

Errors mostly look like:

** (Protocol.UndefinedError) protocol Enumerable not implemented for {:ok, [{"776f3ab8-8395-4ddd-9645-a367de9e7231", nil}, {"17e4819a-4a2c-40a3-8c09-e7aece7932d4", nil}]}. This protocol is implemented for: Ecto.Adapters.SQL.Stream, Postgrex.Stream, DBConnection.Stream, DBConnection.PrepareStream, Timex.Interval, StreamData, HashSet, Range, Map, Function, List, Stream, Date.Range, HashDict, GenEvent.Stream, MapSet, File.Stream, IO.Stream

Are there any expected breaking changes in 3.2.0? (I couldn't find any listed in docs or issues)

Do you have a reproduction case (& code) I can use to check this? We don't do anything special in terms of files, just use the stdlib stuff - so there's probably something wrong with the file somehow, likely due to crashing (maybe the file is half written?).

I cannot reproduce this, but it's because you are using transactions (many of them) at the same time. This is more a logic issue rather than a bug in the LRW policy, because it uses transactions too. You're blocking up the transactions message box. I'd be interested if you actually need transactions, and what your real use case is that's causing this to happen?

I thought that if you have two transactions with different keys:

• Cachex.transaction(cache, [ "key1" ], fn(state) ->
• Cachex.transaction(cache, [ "key2" ], fn(state) ->

then these can run in parallel. In reality this is not the case.

Here's a failing test for Cachex.Actions.TransactionTest:

  test "transactions are independent by key" do
    # create a test cache
    cache = Helper.create_cache([ transactional: true ])

    # spawn a transaction to increment a key
    spawn(fn ->
      Cachex.transaction(cache, [ "key" ], fn(state) ->
        :timer.sleep(50)
        Cachex.incr(state, "key")
      end)
    end)

    # wait for the spawns to happen
    :timer.sleep(10)

    duration_with_result = :timer.tc(fn ->
      Cachex.transaction(cache, [ "other-key" ], fn(state) ->
        assert {:ok, 1} = Cachex.incr(cache, "other-key")
      end)
    end)
    duration_in_millis = duration_with_result |> elem(0) |> Kernel./(1_000)

    assert duration_in_millis < 10
  end

Result:

  1) test transactions are independent by key (Cachex.Actions.TransactionTest)
     test/cachex/actions/transaction_test.exs:33
     Assertion with < failed
     code:  assert duration_in_millis < 10
     left:  41.257
     right: 10

Maybe I'm misunderstanding this and this is actually the expected behavior. In that case do you have recommendations how to have per key transactions?

I'm going to close this for now because it's intended, and you have a workaround for your use case. If someone comes here in future, maybe we can consider changing it - but it's complex so I don't want to do it lightly.