profile
viewpoint
Isaac Whitfield whitfin @appcelerator Mountain View, CA https://whitfin.io Fan of all things automated. OSS when applicable. Author of Cachex for Elixir. Senior Software Engineer at Axway. Intelligence wanes without practice.

whitfin/andrest 7

Basic REST protocol implementation for Android.

appcelerator/lexus 2

A small, portable JSON language for querying data.

whitfin/argle 2

Convenient argument shifting because you know you need it

Axway/platform-petstore 1

A sample Swagger PetStore project, integrated with the AMPLIFY Platform

vasil9v/yaml2json.py 1

Convert a YAML file to JSON

whitfin/appddl 1

Extremely small CLI tool used to update agent archives from AppDynamics

vasil9v/xml2json 0

converts well-formed XML into an equivalent JSON representation and back

whitfin/actix-web 0

Actix web is a small, pragmatic, and extremely fast rust web framework.

whitfin/ajv-cli 0

Use 'Another Json Validator' (ajv) from the command line

issue closedwhitfin/cachex

[Proposal] Add put_and_get function

Let's add put_and_get function to mail API of Cachex.

How it should work

Function put_and_get should work similar to put, but return additional element – value:

iex > Cachex.put_and_get(:my_cache, "key", "value")
{:ok, true, "value"} 
iex >

Motivation

In some use cases we use flow like:


def do_something(key) do

{_, value} = Cachex.get(:my_cache, key)

  case value do
     nil -> 
          new_value = some_heavy_computation(key)
          Cachex.put(:my_cache, key, new_value)
          new_value # the important line
    _ -> value
  end

end

Now if you omit the important line, then you get nil on return from do_something. This change is ‘quality of life‘ change, that protect developers against some type of errors.

Impact to existing API

Nope

closed time in 6 days

Koziolek

issue commentwhitfin/cachex

[Proposal] Add put_and_get function

This is already covered by using get_and_update/4. Please see the examples in the docs: https://hexdocs.pm/cachex/Cachex.html#get_and_update/4

Koziolek

comment created time in 6 days

PR closed whitfin/cachex

Add put_and_get action

This is implementation of #258

+166 -3

1 comment

7 changed files

Koziolek

pr closed time in 6 days

issue commentwhitfin/cachex

Keep data on restart

Yes, if you use import/3 and export/2 - you can export and re-import your data after a restart.

jhonathas

comment created time in 6 days

issue closedwhitfin/siphash-java

404 Not Found when try to download lib from Central or sonatype maven repository

Hello! Would be possible to republish the jar artifacts? These 3 links are returning 404 not found

https://oss.sonatype.org/content/repositories/releases/io/whitfin/siphash/2.0/siphash-2.0.jar https://repo1.maven.org/maven2/io/whitfin/siphash/2.0/siphash-2.0.jar https://repo.maven.apache.org/maven2/io/whitfin/siphash/2.0/siphash-2.0.jar

Thanks

closed time in 24 days

volpatorafael

issue commentwhitfin/siphash-java

404 Not Found when try to download lib from Central or sonatype maven repository

Hi @volpatorafael! You should be using 2.0.0, not 2.0. Everything is working as expected :)

volpatorafael

comment created time in 24 days

created tagwhitfin/runiq

tagv1.2.0

An efficient way to filter duplicate lines from input, à la uniq.

created time in a month

release whitfin/runiq

v1.2.0

released time in a month

push eventwhitfin/runiq

Isaac Whitfield

commit sha 49ca62339d3d51cb367da58b5a03c4cb16b97783

Bump to v1.2.0

view details

push time in a month

delete branch whitfin/runiq

delete branch : windows-compat

delete time in a month

push eventwhitfin/runiq

Isaac Whitfield

commit sha 4743e7a61ef751dd69b41fef67b148b9fd6556c5

Migrate to a Rust xxHash implementation

view details

Isaac Whitfield

commit sha 2c5ce8f475bdfe754b9329bdfd520bd2accdd4e1

Enable Windows builds in CI

view details

Isaac Whitfield

commit sha 805137245fa3afb8b307e2f9e6cf22304731bfc1

Update dependency listing

view details

push time in a month

PR merged whitfin/runiq

Migrate to a Rust xxHash implementation for Windows

Fixes #11 and fixes #9.

Hopefully this makes it easier to build on Windows, where uniq doesn't exist - so this library would be more useful.

+41 -14

0 comment

3 changed files

whitfin

pr closed time in a month

issue closedwhitfin/runiq

Suggestion: Use a Rust implementation of xxHash

Depending on a C library complicates the compilation, since it requires that the Clang compiler be present.

xxHash has already been implemented in Rust in the twox-hash crate.

Using it would probably make it easy to publish pre-built binaries for macOS, Linux and Windows, which would improve the usability of runiq :)

closed time in a month

mardukbp

issue closedwhitfin/runiq

Failed to compile xxhash-sys. Windows x64 MSVC 2019

error: failed to run custom build command for xxhash-sys v0.1.0

Caused by: process didn't exit successfully: C:\Users\Admin\AppData\Local\Temp\cargo-installfAw9KD\release\build\xxhash-sys-696d500ada48d2a1\build-script-build (exit code: 101) --- stdout running: "cmake" "C:\Users\Admin\.cargo\registry\src\github.com-1ecc6299db9ec823\xxhash-sys-0.1.0\xxHash/cmake_unofficial" "-G" "Visual Studio 16 2019" "-Thost=x64" "-Ax64" "-DCMAKE_INSTALL_PREFIX=C:\Users\Admin\AppData\Local\Temp\cargo-installfAw9KD\release\build\xxhash-sys-fa8f6d7a1b4e633c\out" "-DCMAKE_C_FLAGS= -DXXH_NAMESPACE=_rust_xxhash_sys -nologo -MD -Brepro" "-DCMAKE_C_FLAGS_RELEASE= -DXXH_NAMESPACE=_rust_xxhash_sys -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS= -nologo -MD -Brepro" "-DCMAKE_CXX_FLAGS_RELEASE= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS= -nologo -MD -Brepro" "-DCMAKE_ASM_FLAGS_RELEASE= -nologo -MD -Brepro" "-DCMAKE_BUILD_TYPE=Release"

--- stderr thread 'main' panicked at ' failed to execute command: The system cannot find the file specified. (os error 2) is cmake not installed?

build script failed, must exit now', C:\Users\Admin.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.44\src\lib.rs:885:5 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

closed time in a month

anotatta

push eventwhitfin/runiq

Isaac Whitfield

commit sha 244f0c6385814fba0879689c5588ad5c0303732a

Update dependency listing

view details

push time in a month

PR opened whitfin/runiq

Migrate to a Rust xxHash implementation for Windows

Hopefully this makes it easier to build on Windows, where uniq doesn't exist - so this library would be more useful.

+41 -13

0 comment

3 changed files

pr created time in a month

create barnchwhitfin/runiq

branch : windows-compat

created branch time in a month

PR closed whitfin/runiq

Add a --count option
+33 -0

3 comments

3 changed files

pfmoore

pr closed time in a month

pull request commentwhitfin/runiq

Add a --count option

See comments in the related issue :) thank you though!

pfmoore

comment created time in a month

issue closedwhitfin/runiq

Add a -c (count) flag?

Would it be possible to add a -c flag to output a count of each unique line, like uniq has? A significant proportion of my usage of uniq is in the form of sort | uniq -c | sort -n, and being able to use runiq to replace that initial pair of commands would be really nice.

closed time in a month

pfmoore

issue commentwhitfin/runiq

Add a -c (count) flag?

Hi!

Sorry, haven't had time to look at this in a while. In this case though, there's really no need to add this. You can do the following:

cat file.txt | runiq | wc -l
pfmoore

comment created time in a month

push eventwhitfin/bytelines

Dan Davison

commit sha 05fc0ace508578b839ae32de8b4b1473fd4cd5ee

Make comments regarding types more accurate Fixes #2

view details

Isaac Whitfield

commit sha 21502d6329902ca848b872ec1162c8dbe0c12018

Merge pull request #3 from dandavison/fix-docs Make comments regarding types more accurate

view details

push time in a month

PR merged whitfin/bytelines

Make comments regarding types more accurate

As discussed. Fixes #2.

+8 -8

0 comment

1 changed file

dandavison

pr closed time in a month

issue closedwhitfin/bytelines

Documentation contains incorrect types

Apologies if I'm not understanding some Rust conventions but it seems to me that that the docs are misleading about the types yielded when iterating over lines. E.g.

// do something with the line, which is &[u8]

whereas really (AIUI) it's Result<&[u8], Error>. The same is true of several of the comments (and thus in the official docs on crates.io).

closed time in a month

dandavison

release whitfin/local-cluster

v1.2.1

released time in a month

created tagwhitfin/local-cluster

tagv1.2.1

Easy local cluster creation for Elixir to aid in unit testing

created time in a month

push eventwhitfin/local-cluster

Isaac Whitfield

commit sha cff546167392027cac0c3eb601d764da45992723

Bump to v1.2.1

view details

push time in a month

PR closed whitfin/local-cluster

Code.compile_string/2 seems to interfere with other ExUnit tests
  • Using LocalCluster.start_nodes/3 with option :files in an umbrella project corrupted the ExUnit test management. The test file was re-compiled and re-executed for each subsequent umbrella app. A warning was emitted for each re-compile: warning: redefining module XXX (current version defined in memory)
  • This change avoids code changes on the master node by moving the compilation to each slave node (assuming the compiler is loaded there).

Note that this PR changes implementation for #8. Will it work for you? Is there a better solution? Please advise.

+2 -9

7 comments

1 changed file

pmenhart

pr closed time in a month

push eventwhitfin/local-cluster

Isaac Whitfield

commit sha 2499231f45980761b824dd8625fda64cb0a14f86

Fix an issue with repeated inclusions

view details

push time in a month

pull request commentwhitfin/local-cluster

Code.compile_string/2 seems to interfere with other ExUnit tests

@pmenhart ok after reviewing this I'm going to accept your change; I have no idea why v1.2 fails - it's something to do with the anonymous functions but I don't see much difference between the stdlib code for v1.2 and v1.3 here, so it feels like just a limitation of v1.2 somehow.

v1.2 still works with this lib, it's just that the anonymous functions do not. I don't think this is a major issue as there are ways around it and very few people are using v1.2 anyway, so I'm just going to merge as is.

pmenhart

comment created time in a month

PullRequestEvent

pull request commentwhitfin/local-cluster

Code.compile_string/2 seems to interfere with other ExUnit tests

Are you including your test files? That's the only thing I can imagine, and that seems quite odd.

I'll take another look at this today and push a patch; let me think about 1.2 for a while.

pmenhart

comment created time in a month

created tagwhitfin/local-cluster

tagv1.2.0

Easy local cluster creation for Elixir to aid in unit testing

created time in a month

release whitfin/local-cluster

v1.2.0

released time in a month

pull request commentwhitfin/local-cluster

Added root app option

@pmenhart yeah it makes sense given that you can use it to exclude; I missed that initially but realised when merging. Thanks for the work! I'm going to push a new version soon with that PR included; sorry again for the delay.

scohen

comment created time in a month

push eventwhitfin/local-cluster

Isaac Whitfield

commit sha 7b5dde88f59229c5252473cdbb9bd699efcf8ce1

Bump to v1.2.0

view details

push time in a month

push eventwhitfin/local-cluster

Isaac Whitfield

commit sha 06c89821c024ef4635f5277eb3979da4cb3884af

Update CI builds

view details

push time in a month

PR closed whitfin/local-cluster

Allow user to override the Application environment

In order to pass a different environment into the remote node when starting it, the user must change their local Application env, start the node, then change it back. For simple cases this works fine, but once tests are running in parallel, this can cause chaos.

This PR adds an :env_override option that allows the user to inject different environmental variables into the remote node. This is especially useful when each group of nodes in a test must start up on a different port or with different app options enabled.

Build passed: https://travis-ci.org/github/whitfin/local-cluster/builds/667949124

+49 -1

2 comments

3 changed files

veedo

pr closed time in a month

pull request commentwhitfin/local-cluster

Allow user to override the Application environment

Regardless, I merged your changes inside https://github.com/whitfin/local-cluster/commit/523545b0bdc7cb077e8afb080d01ed35d4e53a8e - with a tweaking of the naming to environment and a slight touch of reformatting. Thanks for the contribution!

veedo

comment created time in a month

push eventwhitfin/local-cluster

Isaac Whitfield

commit sha 523545b0bdc7cb077e8afb080d01ed35d4e53a8e

Allow environment overrides on child nodes

view details

push time in a month

issue closedwhitfin/local-cluster

umbrella apps config

I see that application configs are passed to spawned nodes here https://github.com/whitfin/local-cluster/blob/f86e6f043c0d58a2ea7bb44f7185a54a3694b949/lib/local_cluster.ex#L70 And looks like it works, but sometimes in my umbrella project some applications are terminating because of missing configs. I have 2 suggestions

  • some umbrella app somehow is getting started before this line is executed https://github.com/whitfin/local-cluster/blob/f86e6f043c0d58a2ea7bb44f7185a54a3694b949/lib/local_cluster.ex#L70

  • some umbrella app is not loaded (and loading later with missing config) https://github.com/whitfin/local-cluster/blob/f86e6f043c0d58a2ea7bb44f7185a54a3694b949/lib/local_cluster.ex#L68

is it possible?

closed time in a month

tim2CF

issue commentwhitfin/local-cluster

umbrella apps config

I think this is related to https://github.com/whitfin/local-cluster/issues/13 which is resolved by #15. It should be fixed in the next release (or current master). I'm going to close this, but feel free to re-open if you still have issues!

tim2CF

comment created time in a month

issue closedwhitfin/local-cluster

Some clarifications

Hello!

Tried this library and got permanent :not_alive error on LocalCluster.start_nodes Also your library test returned this

00:02:37.767 [info]  Protocol 'inet_tcp': register/listen error: econnrefused                                                                                                                                                   
                                                                                                                                                                                                                                
** (MatchError) no match of right hand side value: {:error, {{:shutdown, {:failed_to_start_child, :net_kernel, {:EXIT, :nodistribution}}}, {:child, :undefined, :net_sup_dynamic, {:erl_distribution, :start_link, [[:"manager@1
27.0.0.1"], false]}, :permanent, 1000, :supervisor, [:erl_distribution]}}}                                                                                                                                                      
    test/test_helper.exs:2: (file)                                                                                                                                                                                              
    (elixir) lib/code.ex:813: Code.require_file/2                                                                                                                                                                               
    (elixir) lib/enum.ex:783: Enum."-each/2-lists^foreach/1-0-"/2    

I am not sure - may be this issue is trivial and self obvious for true erlang developers, but for noobs like me it was kind of pain.

Finally I got the reason - there must be epmd daemon running.

So may be reflect this requirements in documentation? It seems mandatory condition so looks logical for me to write couple of lines about it.

closed time in 2 months

webhive

issue commentwhitfin/local-cluster

Some clarifications

I added a small note about this in the README, thanks!

webhive

comment created time in 2 months

PR closed whitfin/local-cluster

Start the environment using Mix.

When starting the nodes the current implementation does not take in account applications deps, which results in application started out of order and configurations wrongly merged. Using Mix allow to start in the correct order with the very same environment that the developer expects.

+1 -10

5 comments

1 changed file

xadhoom

pr closed time in 2 months

pull request commentwhitfin/local-cluster

Start the environment using Mix.

So... I looked this over and I believe it can be resolved by #15, which has been merged. Want to try on latest master?

I'm going to close for now, but feel free to re-open if you think it should be!

xadhoom

comment created time in 2 months

PR closed whitfin/local-cluster

Code.compile_string/2 seems to interfere with other ExUnit tests
  • Using LocalCluster.start_nodes/3 with option :files in an umbrella project corrupted the ExUnit test management. The test file was re-compiled and re-executed for each subsequent umbrella app. A warning was emitted for each re-compile: warning: redefining module XXX (current version defined in memory)
  • This change avoids code changes on the master node by moving the compilation to each slave node (assuming the compiler is loaded there).

Note that this PR changes implementation for #8. Will it work for you? Is there a better solution? Please advise.

+2 -9

3 comments

1 changed file

pmenhart

pr closed time in 2 months

push eventwhitfin/local-cluster

Isaac Whitfield

commit sha 62ce68aa65570d9153332e591303a7d5c578b096

Suppress harmless re-compilation warnings

view details

push time in 2 months

pull request commentwhitfin/local-cluster

Code.compile_string/2 seems to interfere with other ExUnit tests

On second look, there's actually no issue here - the warning is just because the string is being compiled repeatedly. This is not an issue though because you will already have loaded the same module anyway, so the warning is redundant. I don't think we need to "fix" this. The warning is annoying, but it shouldn't be harming anything.

pmenhart

comment created time in 2 months

push eventwhitfin/local-cluster

Isaac Whitfield

commit sha b75432ea1da67e81d59ddf1128355003b867da84

Minor refactoring and documentation tweaks

view details

push time in 2 months

PR closed whitfin/local-cluster

Reviewers
Ensure all configuration merged before starts loaded applications in child nodes

Please review df9be34 mainly, other commits are unrelated to main issue.

The current implementation copies application config (Application.put_env/4) and starts loaded applications (Application.ensure_all_started/2) simultaneously. But it can start an application before copying its configuration if other application starts it.

So in this PR, I modified code to starts applications after all configuration merged.

+74 -40

2 comments

4 changed files

ishikawa

pr closed time in 2 months

pull request commentwhitfin/local-cluster

Ensure all configuration merged before starts loaded applications in child nodes

I believe this is fixed with the merge of #15, so I'm going to close it for now. Please test with the latest master and verify that it is; if not let me know and we can re-open!

ishikawa

comment created time in 2 months

PR closed whitfin/local-cluster

Added root app option

Prior to this, applications were started in a non-deterministic order, which could lead to misconfiguration or applications not booting. Now you can specify a root app that will be started prior to all the other applications. Due to the way that applications are loaded, specifying this single app should be enough to boot correctly.

+44 -30

6 comments

1 changed file

scohen

pr closed time in 2 months

pull request commentwhitfin/local-cluster

Added option to specify applications to start on cluster nodes

I've merged this in the meantime, since the naming doesn't block the merge. I'm likely going to tweak the option to :applications rather than :app_names, but the rest looks good!

pmenhart

comment created time in 2 months

push eventwhitfin/local-cluster

Peter Menhart

commit sha dcaf7dedbbbc0bd9082e2cb3d2fe32f4dabda760

Added option to specify applications to start on cluster nodes (#15)

view details

push time in 2 months

PR merged whitfin/local-cluster

Added option to specify applications to start on cluster nodes
  • Default behaviour: start_nodes/3 starts all applications that are loaded on the master node. There are situations where some apps are not desirable on slave nodes, or where their start order is causing problems.
  • New option :app_names accepts an ordered list of applications to be started. Very useful for test scenarios where only a subset of apps should be copied over, or where the order of applications starting is critical.
  • Note that each application is started via Application.ensure_all_started/2, i.e. with all dependencies.
+53 -1

1 comment

3 changed files

pmenhart

pr closed time in 2 months

push eventwhitfin/local-cluster

Bernard Duggan

commit sha 9d5d03f982c9058c8212c754a226f5f893b61a55

Fix param for setting remote log level (#12)

view details

push time in 2 months

PR merged whitfin/local-cluster

Fix param for setting remote log level

The remote log level wasn't being set correctly due to the subtleties of how Elixir handles keyword lists.

[a: :b] gets interpreted as [{:a, :b}], but the args list needs [[{:a, :b}]], since the keyword list is just one of multiple possible arguments. This patch gets the log level set correctly.

+1 -1

0 comment

1 changed file

bernardd

pr closed time in 2 months

PR closed whitfin/local-cluster

Allow skipping the loading of certain applications on remote nodes

This fixes an issue (specifically with propcheck) where the other nodes would fail to boot. Propcheck tries to access it's context dets file at boot and having multiple instances launch at the same time causes errors like:

06:42:50.228 [info]  Application propcheck exited:
   PropCheck.App.start(:normal, []) returned an error: shutdown: failed
to start child: PropCheck.CounterStrike
    ** (EXIT) an exception was raised:
        ** (MatchError) no match of right hand side value: {:error,{:not_a_dets_file, '_build/propcheck.ctex'}}
            (propcheck 1.2.0) lib/counterstrike.ex:53:PropCheck.CounterStrike.init/1
            (stdlib 3.9.2) gen_server.erl:374: :gen_server.init_it/2
            (stdlib 3.9.2) gen_server.erl:342: :gen_server.init_it/6
            (stdlib 3.9.2) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

The solution for this is to allow local cluster to skip loading certain dependencies when booting the other nodes.

+22 -1

3 comments

2 changed files

jeffutter

pr closed time in 2 months

pull request commentwhitfin/local-cluster

Allow skipping the loading of certain applications on remote nodes

Yes, I'm going to close this in favour of #15 - which I am intending to merge in some form once I get some clarification on another thread. Thanks for the PR though!

jeffutter

comment created time in 2 months

pull request commentwhitfin/local-cluster

Code.compile_string/2 seems to interfere with other ExUnit tests

Seems fine to me, but the 1.2 build is curious - any ideas why?

pmenhart

comment created time in 2 months

pull request commentwhitfin/local-cluster

Added root app option

Based on code alone, IMO, #15 is indeed the way to go seeing as you can accomplish what this PR does already, with extras. @scohen I believe your "root" app would simply be the first app in the multiple apps list in #15, thus making this PR redundant in the grand scheme of things.

However I don't really understand the multiple apps thing either; what use case would require starting multiple apps in a specific order where you wouldn't only spawn the root app, as @scohen says?

@keathley I'm not sure what you have included in that fork, but if it's only a few of these PRs and you've been using them in production, I'm happy to try get it merged back here and push a release.

scohen

comment created time in 2 months

Pull request review commentwhitfin/local-cluster

Added option to specify applications to start on cluster nodes

 After calling `start_nodes/2`, you will receive a list of node names you can the to communicate with via RPC or however you'd like. Although they're automatically cleaned up when the calling process dies, you can manually stop nodes as well to test disconnection. +`start_nodes/3` accepts list of options as an additional parameter.+Two options are currently supported: `:app_names` and `:files`, as described below.++Use option `:app_names` with an ordered list of application names for scenarios

I think I also prefer :include rather than :files, thoughts?

pmenhart

comment created time in 2 months

Pull request review commentwhitfin/local-cluster

Added option to specify applications to start on cluster nodes

 After calling `start_nodes/2`, you will receive a list of node names you can the to communicate with via RPC or however you'd like. Although they're automatically cleaned up when the calling process dies, you can manually stop nodes as well to test disconnection. +`start_nodes/3` accepts list of options as an additional parameter.+Two options are currently supported: `:app_names` and `:files`, as described below.++Use option `:app_names` with an ordered list of application names for scenarios

I would prefer :applications, to be in line with other places such a list is used.

pmenhart

comment created time in 2 months

PullRequestReviewEvent
PullRequestReviewEvent

pull request commentwhitfin/local-cluster

Added root app option

Hi, yes it's active - sorta. I have little time for Elixir these days - and this project is low on a list of already low priorities, particularly given the global situation.

I'll take a look at this PR and the linked PR though, and see if we can't get them in soon :)

scohen

comment created time in 2 months

pull request commentwhitfin/local-cluster

Allow user to override the Application environment

I'm not sure I understand the use case; why would your test environment have different options to the environment you're running your tests under? What's the use case that you can't just change them in your local env?

veedo

comment created time in 2 months

PR closed whitfin/cachex

Add failing test for ttl opt for fetch

Hi, I've added this test in order to try to illustrate my thoughts on how the API was going to work. I can understand if there are reasons why it shouldn't work that way but for my use case and my level of understanding, it would be a really useful addition.

I'm afraid I would not be confident in trying change things so that this test passed and I'm not sure if that would be desirable anyway.

Please close this PR if this isn't a reasonably direction to go in.

Aimed at helping: #253


The 'ttl: 1' is not respected and so the entry doesn't expire so we don't get the "used" value back from the fallback on the fetch.

The assert on key10 works but because the ttl isn't set rather than because the ttl is long enough that it shouldn't have expired.

+17 -0

1 comment

1 changed file

michaeljones

pr closed time in 2 months

pull request commentwhitfin/cachex

Add failing test for ttl opt for fetch

It is not expected that this works; the ttl option is not supported with fetch. See here.

michaeljones

comment created time in 2 months

startedRiotGames/leaguedirector

started time in 2 months

created tagwhitfin/cachex

tagv3.3.0

A powerful caching library for Elixir with support for transactions, fallbacks and expirations

created time in 3 months

release whitfin/cachex

v3.3.0

released time in 3 months

push eventwhitfin/cachex

Isaac Whitfield

commit sha 2319de984879cd652c4a44113676d34c49d7da09

Bump to v3.3.0

view details

push time in 3 months

push eventwhitfin/cachex

Isaac Whitfield

commit sha 5699a4346d6ac0b73fa8c93a14cfffb8f1606ba1

Minor tweaks to warmer startup

view details

push time in 3 months

push eventwhitfin/cachex

nico piderman

commit sha 9fdbe7912b6467b48f0be8da66fe79df4845cb2e

Make Cachex.Warmer block on start (#244)

view details

push time in 3 months

PR merged whitfin/cachex

update Cachex.Warmer to block startup during first execution

Closes #242

FWIW, I have been running this fork in production for a few weeks now without any issues.

+17 -18

2 comments

2 changed files

sneako

pr closed time in 3 months

issue closedwhitfin/cachex

Blocking Preemptive Warmer

I have an application in which I store some externally managed configuration in Cachex. Without this configuration, the application is completely unusable, and should not accept any reqeuests.

I use a Preemptive Warmer to hydrate this cache, but since the warmer is executed asynchronously, my application finishes booting, and begins accepting requests before the warmer has finished executing.

Would you be open to a PR adding an option to use Cachex.Warmer, maybe something like block_on_startup: true, which would cause the execute callback to be executed during init?

Thanks for the amazing library!

closed time in 3 months

sneako

issue commentwhitfin/cachex

OOM due to Cachex.Policy.LRW not being able to process messages in time

I couldn't get it to happen, but I'll check again in case I missed anything.

As for the usage of transactions, we want to make sure the fallback function doesn't get called concurrently for the same key.

This is automatically handled for you; if two calls happen for the same key, the first one will do the fetch and the second call will just await the result of the first one. You shouldn't have to do it in a transaction; if you see behaviour differently to this, there's a bug somewhere.

ananthakumaran

comment created time in 3 months

pull request commentwhitfin/cachex

add block_on_startup option to Cachex.Warmer

Hi @sneako, I'm sorry it took so long to get around to this!

There are two things I want to discuss here; the first is that I believe this should be considered a bug, and as such your optional change should be our only behaviour. As it stands in the latest version, applications can go live without potentially vital data (as is your case). This is definitely an oversight. It also means that if your warmer is failing, your application might go live and simply never have cached data (since a crashing warmer would not stop your application starting). This also feels like a mistake.

I'm interested in your thoughts, seeing as you hit it naturally. I think we should perhaps just avoid adding an option, and just change the current behaviour to block. I can't think of a compelling argument against this; the only negative I can think of is that startup time might be longer, but you're trading that for safety so it's always worth it. I'm pretty convinced this is the route we should take, but I want to make sure I'm not missing anything before we do so. One thing I'm concerned about is people using the current version who do not update or use an option, and they don't realise this is happening (or has the potential to happen).

If we do decide to have an option, I don't like that this PR places it on the Warmer options itself. I'd much rather add an option to the warmer record passed to the cache startup, so you could start the same Warmer with/without blocking. It feels like a more logical place to put it, I think. I'm also not a huge fan of the option name, I'd prefer something simpler like :blocking, but that's kinda nitpick.

Either way, good report! Depending what you think, I'd like to get a solution to this merged and shipped in a minor version. If we decide to go without the option, it should be a minor change that we can get in quickly. I'll await your response for a while; but if you're busy there's a chance I'll grab this at some point to bundle into a release I'm planning shortly for compatibility purposes.

sneako

comment created time in 3 months

PR closed whitfin/cachex

Ttl doc update

As per discussion in #195, minor enhancements to docs to clarify limitations of :ttl option.

+19 -0

1 comment

2 changed files

mindok

pr closed time in 3 months

pull request commentwhitfin/cachex

Ttl doc update

I'm going to close this as I lifted it into the current state of https://github.com/whitfin/cachex/blob/master/docs/features/ttl-implementation.md - let me know if that looks good to you!

I'm not going to change the API docs themselves just yet; I need to figure out what I want first. I'm considering removing the option entirely in the next major version of Cachex (whenever that may be), since I think it causes confusion and isn't worth the slight performance gain you get from it. It would leave us with only two options; either a cache wide default, or expire/4. Much better, I think?

mindok

comment created time in 3 months

push eventwhitfin/cachex

Isaac Whitfield

commit sha 7984a07b58b99c0fd37255b9c0780b4eb7ca0a92

Further documentation improvements around expirations

view details

push time in 3 months

PR closed whitfin/cachex

Improve doc examples

Just trying to improve a bit the lib's documentation for new users.

+35 -3

2 comments

2 changed files

tiagopog

pr closed time in 3 months

pull request commentwhitfin/cachex

Improve doc examples

I've taken your changes from this up in some other changes I've made to the README, so I'm going to close this for now. Thanks for the contributions!

tiagopog

comment created time in 3 months

issue closedwhitfin/cachex

Disable cache for test, or the ability to clear it in async modes

When using async tests with a shared Postgres connection, clearing a cache key in the test code does not clear the cache key in the server-side.

If you would inspect the cache in the test, it will be empty but when you inspect the key on the app side it isn't.

I would like to disable the cachex for the test, or clear the cache key in the setup block only as described the the Cachex.reset(:organisations) does not work.

This is only the case for Async tests.

I've read https://github.com/whitfin/cachex/issues/213 i think the same issue applies here, only the question was not relayed properly.

Thanks!

closed time in 3 months

reneweteling

issue commentwhitfin/cachex

Disable cache for test, or the ability to clear it in async modes

Yes, this is the same question as the other thread. I read it again and thought my final response seemed a little harsh (not quite the word I want to use), but then I read it once again and I believe my response is actually the same.

Logically, the only way this can be an issue is if you have two tests running at the same time, both operating on the same cache key. In this scenario, the flaw lies in your test code. The Cachex.reset/1 function already does what you are asking for; and yes it does work, because it's what I use in the library tests themselves (it's why this function exists). The root cause of your issue is that you have one test clearing your key, whilst the other test is putting it back.

I'm not sure if this issue is asking for changes, but there is no way to stop this happening than to either:

  • Use a different key in each test
  • Run your tests sequentially (non-async)

This isn't something that can be "fixed" on the side of Cachex, because the logic issue is inside your test setup. The same concept applies to your question about disabling the cache for your tests; I'm not the person starting the cache - it's in your code somewhere. In the place that is done, you can check for the Mix environment and disable the cache accordingly.

Let me know if this makes sense; I'm going to close this in the meantime!

reneweteling

comment created time in 3 months

issue closedwhitfin/cachex

Cachex guarantees

Hello, I'm having some trouble understanding some guarantees of Cachex as a cache mechanism for databases (using Ecto with Postgres in my case).

For example, let's say I want to add a new value to the database and at the same time add it to the cache list, something like this:

{:ok, _} =  MyRepo.insert(value)

Cachex.get_and_update(:my_cache, "key", fn
  (nil) -> {:ignore, key}
  (list) -> {:commit, [value | list]}
end)

What I don't understand is if functions like get_and_update are guaranteed to succeed. If that is not guaranteed, it means that in the above example I can insert the value to the database, and then get_and_update fails for some reason making my cache outdated as it will never contain the value when I use get or fetch.

I guess I can make sure I don't have that problem by using an Ecto.transaction with Multi and adding the get_and_update as the last step of Multi. that way the database write would rollback if Cachex call failed.

Is that necessary or I can safely use something like the above example as long as my fn for the third parameter is not broken?

closed time in 3 months

sezaru

issue commentwhitfin/cachex

Cachex guarantees

The best way to say this is that it's undefined. In the current codebase, these things are basically guaranteed - but you should react to the result that is returned and handle it accordingly. This ensures that you are protected in case Cachex ever changes the underlying implementation such that it's no longer guaranteed.

Doing what you said with transactions makes a lot of sense. Another pattern is to avoid manually updating the cache, and instead use a warmer or a fallback mechanism with a TTL. LMK if you need some examples!

sezaru

comment created time in 3 months

push eventwhitfin/cachex

Isaac Whitfield

commit sha 2cceb53658a4eb86d6e5249939a495a450dedfcc

Revert AppVeyor to v1.9 as a test

view details

push time in 3 months

push eventwhitfin/cachex

Isaac Whitfield

commit sha 2b85ca3b87bcdf03e7a93c13afd79e4abee7a748

Update version compatibility table

view details

push time in 3 months

push eventwhitfin/cachex

Isaac Whitfield

commit sha d285086f052552621f471150c97bfa0321714b47

Update CI versions to latest

view details

push time in 3 months

issue closedwhitfin/cachex

Question regarding Supervisor.Spec

I was reading issue 148 and I was wondering if/when this project might be updated to use the new child specifications (Elixir 1.5+)?

In issue 148 @whitfin mentions what I assume is to refactor closer to the Supervisor.Spec being removed from the Elixir library, i.e. Elixir 2.0+.

I recall a talk by José that Elixir would be a long time before that happens. This is good, as Supervisor.Spec and worker/3 will continue to work.

This is more of a question if that refactor is in the roadmap, or if this is a desired refactor for others to do?

... Forgot to add thanks for the cool project.

closed time in 3 months

micahrye

issue commentwhitfin/cachex

Question regarding Supervisor.Spec

Hey everyone! Sorry it took forever (fairly distracted with current situations) but I've updated the signatures based on the advice from José above, and everything looks good. I updated the documentation too, so it should be more familiar. Please let me know if anything is encouraging a bad pattern; I haven't written much Elixir since the spec changes so I want to make sure I'm not portraying anything incorrectly.

I tested the changes with a few dependent packages from Hex and I'm pretty sure everything is fine in terms of backwards compatibility too, so should be good! I'm going to close this, but please do re-open if anything needs improvement. Sorry again for the delay!

micahrye

comment created time in 3 months

push eventwhitfin/cachex

Isaac Whitfield

commit sha f64fc415e6a518d83aa38d0a803110f3073fc9ee

Update to 22.3 OTP versions in CI

view details

Isaac Whitfield

commit sha 46f8bf19ab33cd8346ac806a26e6c105069be780

Update instructions and Supervisor compatibility

view details

push time in 3 months

issue closedwhitfin/cachex

Breaking changes in 3.20

I'm trying to upgrade from cachex 3.1.3 to 3.2.0 and some tests are failing

Errors mostly look like:

** (Protocol.UndefinedError) protocol Enumerable not implemented for {:ok, [{"776f3ab8-8395-4ddd-9645-a367de9e7231", nil}, {"17e4819a-4a2c-40a3-8c09-e7aece7932d4", nil}]}. This protocol is implemented for: Ecto.Adapters.SQL.Stream, Postgrex.Stream, DBConnection.Stream, DBConnection.PrepareStream, Timex.Interval, StreamData, HashSet, Range, Map, Function, List, Stream, Date.Range, HashDict, GenEvent.Stream, MapSet, File.Stream, IO.Stream

Are there any expected breaking changes in 3.2.0? (I couldn't find any listed in docs or issues)

closed time in 3 months

axelson

issue commentwhitfin/cachex

Sometimes cachex.load() says file unreachable, when it very much is reacahble

Do you have a reproduction case (& code) I can use to check this? We don't do anything special in terms of files, just use the stdlib stuff - so there's probably something wrong with the file somehow, likely due to crashing (maybe the file is half written?).

krainboltgreene

comment created time in 3 months

issue commentwhitfin/cachex

OOM due to Cachex.Policy.LRW not being able to process messages in time

I cannot reproduce this, but it's because you are using transactions (many of them) at the same time. This is more a logic issue rather than a bug in the LRW policy, because it uses transactions too. You're blocking up the transactions message box. I'd be interested if you actually need transactions, and what your real use case is that's causing this to happen?

ananthakumaran

comment created time in 3 months

issue closedwhitfin/cachex

Transactions are sharing a lock even when using different keys

I thought that if you have two transactions with different keys:

  • Cachex.transaction(cache, [ "key1" ], fn(state) ->
  • Cachex.transaction(cache, [ "key2" ], fn(state) ->

then these can run in parallel. In reality this is not the case.

Here's a failing test for Cachex.Actions.TransactionTest:

  test "transactions are independent by key" do
    # create a test cache
    cache = Helper.create_cache([ transactional: true ])

    # spawn a transaction to increment a key
    spawn(fn ->
      Cachex.transaction(cache, [ "key" ], fn(state) ->
        :timer.sleep(50)
        Cachex.incr(state, "key")
      end)
    end)

    # wait for the spawns to happen
    :timer.sleep(10)

    duration_with_result = :timer.tc(fn ->
      Cachex.transaction(cache, [ "other-key" ], fn(state) ->
        assert {:ok, 1} = Cachex.incr(cache, "other-key")
      end)
    end)
    duration_in_millis = duration_with_result |> elem(0) |> Kernel./(1_000)

    assert duration_in_millis < 10
  end

Result:

  1) test transactions are independent by key (Cachex.Actions.TransactionTest)
     test/cachex/actions/transaction_test.exs:33
     Assertion with < failed
     code:  assert duration_in_millis < 10
     left:  41.257
     right: 10

Maybe I'm misunderstanding this and this is actually the expected behavior. In that case do you have recommendations how to have per key transactions?

closed time in 3 months

indrekj

issue commentwhitfin/cachex

Transactions are sharing a lock even when using different keys

I'm going to close this for now because it's intended, and you have a workaround for your use case. If someone comes here in future, maybe we can consider changing it - but it's complex so I don't want to do it lightly.

indrekj

comment created time in 3 months

more