profile
viewpoint
Nikita Pekin indiv0 /dev/null http://nikitapek.in I alternate between spaces and tabs every other line

tomassedovic/tcod-rs 203

Rust bindings for libtcod 1.6.3 (the Doryen library/roguelike toolkit)

indiv0/colonize 103

A Dwarf Fortress/Rimworld-like game written in Rust

indiv0/lazycell 43

Rust library providing a lazily filled Cell

indiv0/paste-rs 17

A paste tool written entirely in Rust, and powered by nickel.rs.

indiv0/wolfram-alpha-rs 8

Rust bindings for the Wolfram|Alpha web API

indiv0/smexybot 7

A bot for Discord, written in Rust using the serenity-rs library.

indiv0/launchcraft 4

A python script to automate the installation of my preferred minecraft setup.

indiv0/xkcd-rs 4

Rust bindings for the XKCD web API

indiv0/tcod_window 3

A TCOD back-end for the Piston game engine

indiv0/conductor-rs 2

Netflix Conductor client for Rust

startedjeremyandrews/goose

started time in 18 days

issue commentNetflix/conductor

Workflow Removed from Redis but not Marked as Archived in ES

Hmm. Interesting. That's a good question. Sounds like there's a hole in my theory. I'll investigate deeper.

indiv0

comment created time in 19 days

fork indiv0/BoxGame

Another C#/XNA game I am working on.

fork in 22 days

startedxacrimon/hyphae

started time in 23 days

issue openedNetflix/conductor

Workflow Removed from Redis but not Marked as Archived in ES

Hello!

I have a custom WorkflowArchiverModule that searches for workflows to archive with IndexDAO::searchArchivableWorkflows and then archives them with ExecutionDAO::removeWorkflow(workflowId, true).

What this should do is remove the workflow from redis and mark it as archived: true in ES. Usually this works fine.

Sometimes, this process fails, and the workflow is removed from redis but not marked as archived: true in ES. I believe this happens when ES puts the indices into a read_only_allow_delete state if we're hitting high disk usage watermarks.

To mitigate this, I'm going to avoid running the workflow archiver if our ES indices are in a read-only state, but I was wondering how to recover from this failure when ES comes back online?

If a workflow is stuck in this state and ES comes back to read/write, then the next time the workflow archiver runs it'll get that workflow from IndexDAO::searchArchivableWorkflows but ExecutionDAO::removeWorkflow(workflowId, true) will fail with:

287211533 [workflow-archiver-0] ERROR com.netflix.conductor.core.orchestration.ExecutionDAOFacade  - No such workflow found by id: 50049693-2e5c-4ce9-9007-ed33c9830b78
287211533 [workflow-archiver-0] WARN  com.cpaas.platform.WorkflowArchiverModule$WorkflowArchiver  - Error archiving workflow: 50049693-2e5c-4ce9-9007-ed33c9830b78
com.netflix.conductor.core.execution.ApplicationException: BACKEND_ERROR - Error removing workflow: 50049693-2e5c-4ce9-9007-ed33c9830b78
        at com.netflix.conductor.core.orchestration.ExecutionDAOFacade.removeWorkflow(ExecutionDAOFacade.java:204)
        at com.netflix.conductor.service.ExecutionService.removeWorkflow(ExecutionService.java:342)
        at com.cpaas.platform.WorkflowArchiverModule$WorkflowArchiver.archiveWorkflow(WorkflowArchiverModule.java:132)
        at com.cpaas.platform.WorkflowArchiverModule$WorkflowArchiver.searchAndArchiveWorkflows(WorkflowArchiverModule.java:117)
        at com.cpaas.platform.WorkflowArchiverModule$WorkflowArchiver.lambda$init$0(WorkflowArchiverModule.java:92)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.netflix.conductor.core.execution.ApplicationException: No such workflow found by id: 50049693-2e5c-4ce9-9007-ed33c9830b78
        at com.netflix.conductor.core.orchestration.ExecutionDAOFacade.getWorkflowById(ExecutionDAOFacade.java:80)
        at com.netflix.conductor.core.orchestration.ExecutionDAOFacade.removeWorkflow(ExecutionDAOFacade.java:182)

Is there a way to mark this workflow as archived: true from within a conductor-server module, skipping the redis deletion? If not, could we perhaps add that functionality?

created time in a month

issue commentNetflix/conductor

This Exception where IndexDAO gave ExecutionDAO something that is not in DB yet it say can not find in ES index...

I seem to be observing the same behaviour as well. @s50600822 do you know what caused it?

s50600822

comment created time in a month

more