profile
viewpoint

pierreis/erlang-xxhash 22

Erlang wrapper for xxHash.

pierreis/webleak 1

🌊 Get notified when your e-mail or password leaks on the internet. Powered by Webtask.

pierreis/badger 0

Fast key-value DB in Go.

startedjeremydaly/dynamodb-toolbox

started time in a day

issue commenthanwen/go-fuse

Add optional attr shortcut for readdirplus

Thanks for the suggestion, I'll look into it. Unfortunately, we are getting the information from an upstream service that we cannot really alter.

pierreis

comment created time in 7 days

issue commenthanwen/go-fuse

Add optional attr shortcut for readdirplus

Example:

When developing a loopback file system, the ioutil.ReadDir for example returns a list of os.FileInfo. In most cases, this is sufficient to answer the lookup query. Not being able to pass the information on lookup means potentially costly stat syscalls in lookup that would be unnecessary if we could pass this information right from the readdir.

pierreis

comment created time in 7 days

issue openedhanwen/go-fuse

Add optional shortcut for readdirplus

Currently, the readdirplus implementation provides no simple way to pass attrs in the DirStream. This consequently requires potentially costly lookup calls that could be avoided altogether.

I am thinking about creating a merge request that would:

  • Add an optional Attr field in DirEntry
  • Update Readdirplus logic so that this field is used if provided, otherwise a lookup is performed

Do you think such a pull could be accepted?

created time in 7 days

issue commentjeremydaly/dynamodb-toolbox

Simplified transaction handling

Sounds good. I think I may be able to have a first version working shortly. Other than this, do you think the transaction should be a class (in which case it would be missing the new keyword in your example)?

It seems to me that exporting a function transaction could make more sense, considering that there is not a lot to encapsulate here -- transactions are mostly one shot.

jeremydaly

comment created time in 17 days

issue commentjeremydaly/dynamodb-toolbox

Simplified transaction handling

@jeremydaly -- I am trying to see how I can help implementing this. The main issue with the API you provide and the current code is that if autoExecute is enabled, the nested operations will be executed. As these operations are built independently, I see no obvious way to instruct these to not execute if set in the context of a transaction.

The options I see:

  • Either request the user to explicitly set options.execute to false explicitly on each operation if not set globally, and raise an exception at runtime if not set. This seems unnecessarily verbose.

  • Either change the transaction API to be more like the following, which makes the API also quite verbose

let txn = new Transaction({
  ReturnConsumedCapacity: ‘TOTAL’
});
tx.update(Model1, someItem1);
tx.delete(Model2, someItem2);
const result = await tx.execute();
  • Either provide a transactional wrapper to each of these methods: Model1.transactionalUpdate(someItem)

In any case, none of these solution are very appealing to me. Did you have a thought on how you would implement this?

Regarding transaction auto-splitting, I don't think this should be the default, as it inherently breaks all consistency guarantees of transactions. There may be a case for this, but it should definitely be a setting for which the default is to throw an error if the limit is exceeded.

jeremydaly

comment created time in 17 days

issue commentdavidmdm/myzod

V1.0.0 release

Hi David,

Sure. I think it is okay to release.

Returning an error array is an important feature (that actually would help me a lot), but I feel like there is no strong rationale here to do it before 1.0 rather than in a future version 1.1.

Best,

— Pierre

On May 8, 2020, at 01:04, David Desmarais-Michaud notifications@github.com wrote:

@pierreishttps://github.com/pierreis ,

I have taken a little more than a week off from actively developing myzod and am now recharged. I've been wondering if it would be a good idea to release version 1.0.0

The only feature I haven't done that people have asked for was to have every error fields on ValidationErrors. ie:

const personSchema = mz.object({ name: mz.string(), lastname: mz.string() }); const err = personSchema.try({ name: null });

// Possibly fields would contain the failed keys and their error message

err.fields = {
  name: 'expected type string but got null',
  lastname: 'expected type string but got undefined',
}

Instead of simply throwing on the first error encountered. This would make it useful for Form Validations on the front-end. Custom default error messages would need to go hand in hand with this.

So my question is, should this feature be implemented and if yes before v1.0.0?

My feeling is that releasing a non-alpha version might help with adoption and provide more feedback and feature requests.

Just looking for thoughts, and you're my number one collaborator. Thanks for your efforts and being active within myzod.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub<https://github.com/davidmdm/myzod/issues/13>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AABRZROV3F6G24SSR752NKLRQM46TANCNFSM4M3XIKNA>.
davidmdm

comment created time in 19 days

issue commentdavidmdm/myzod

Property 'shape' does not exist on type 'ObjectType<{ ... }>'

That should fix the issue. ObjectType.shape was introduced in v1.0.0-alpha.7.

kaedub

comment created time in 19 days

issue commentdavidmdm/myzod

Property 'shape' does not exist on type 'ObjectType<{ ... }>'

I cannot reproduce the error:

import * as z from "myzod";
const o = z.object({ test: z.string() });
console.log(o.shape()); // => correctly returns the shape

Which version of the library are you using?

kaedub

comment created time in 19 days

pull request commentnecolas/react-native-web

Enable window/body based scrolling for all ScrollViews

That sounds like a fantastic feature, thanks! Any way to help?

EyMaddis

comment created time in 21 days

issue openedvriad/zod

Type inference not working properly

Zod version: 1.5.0 TS version: 3.8.3

The type inference does not seem to be working properly, making all values of objects to be optional by default.

Let me take this code derived from the readme:

import * as z from "zod";

const dogSchema = z.object({
  name: z.string(),
  neutered: z.boolean(),
});

type T = z.TypeOf<typeof dogSchema>;  // -> { name?: string, neutered?: boolean }

const dog: T = {};  // -> Compiles without error

created time in 24 days

issue commentvriad/zod

Implement object

I am not sure to understand what you mean by proper getter / setter? This seems to be exactly what defineProperty is doing.

pierreis

comment created time in a month

Pull request review commentdavidmdm/myzod

default v2

 export class NullableType<T extends AnyType> extends Type<Infer<T> | null> imple   and<K extends AnyType>(schema: K): IntersectionType<this, K> {     return new IntersectionType(this, schema);   }-  default(value: Nullable<Infer<T>> | (() => Nullable<Infer<T>>)) {-    return new NullableType(this.schema, value);+}++// type X = {+//   a: DefaultableType<StringType>;+//   b: DefaultableType<NumberType>;+// };++type DS = DefaultableType<StringType>;++type R = DS extends Type<any> ? (DS extends Type<infer K> ? K : DS) : any;

I cannot really see a use-case from my point of view, so I wouldn't mind.

davidmdm

comment created time in a month

Pull request review commentdavidmdm/myzod

default v2

 export class NullableType<T extends AnyType> extends Type<Infer<T> | null> imple   and<K extends AnyType>(schema: K): IntersectionType<this, K> {     return new IntersectionType(this, schema);   }-  default(value: Nullable<Infer<T>> | (() => Nullable<Infer<T>>)) {-    return new NullableType(this.schema, value);+}++// type X = {+//   a: DefaultableType<StringType>;+//   b: DefaultableType<NumberType>;+// };++type DS = DefaultableType<StringType>;++type R = DS extends Type<any> ? (DS extends Type<infer K> ? K : DS) : any;

Sounds good. Debugging advanced types in TypeScript is hellish... I still have no clue.

davidmdm

comment created time in a month

Pull request review commentdavidmdm/myzod

default v2

 export class NullableType<T extends AnyType> extends Type<Infer<T> | null> imple   and<K extends AnyType>(schema: K): IntersectionType<this, K> {     return new IntersectionType(this, schema);   }-  default(value: Nullable<Infer<T>> | (() => Nullable<Infer<T>>)) {-    return new NullableType(this.schema, value);+}++// type X = {+//   a: DefaultableType<StringType>;+//   b: DefaultableType<NumberType>;+// };++type DS = DefaultableType<StringType>;++type R = DS extends Type<any> ? (DS extends Type<infer K> ? K : DS) : any;

I am not sure. That is quite disturbing.

Let me check.

davidmdm

comment created time in a month

issue commentdavidmdm/myzod

Default values

Agree on the last point. Probably not a big deal ; IMO it falls in the "sure you can do it, but not a big deal" category.

pierreis

comment created time in a month

issue commentdavidmdm/myzod

Default values

In the long run, I might try an implementation based on a DefaultedType<T> class, similar to what you have done for Optional and Nullable types, that would likely solve each point.

pierreis

comment created time in a month

issue commentdavidmdm/myzod

Default values

I don't think it is too complicated to do it in both cases. I definitely think that OptionalType should peek into the nested schema -- this would also solve the const schema = z.object({ a: z.string().optional().default('hello') }).default({}); case, killing two birds with one stone.

I don't necessarily agree with you on the fact that I may pass the default for nested values right inside the object, as it breaks encapsulation. If I have an addressSchema nested inside some aserSchema, I would need to do the following:

const userSchema = z.object({
  id: z.string(),
  address: addressSchema,
}).default({
  id: uuidv4,
  address: addressSchema.parse(undefined),

While this is technically not impossible, it is not very pretty either. Maybe we could add some parameterless default() method that will return the default value, so the above would be:

const userSchema = z.object({
  id: z.string(),
  address: addressSchema,
}).default({
  id: uuidv4,
  address: addressSchema.default(),

It is still not ideal, but does look a tad more expressive.

pierreis

comment created time in a month

issue commentdavidmdm/myzod

Default values

I think you did miss one comment of mine on the pull request:

const schema = z.object({
  a: z.string().default('hello'),
}).default({});

// Does not compile, because `a` is not optional. Default should imply optional.
const schema = z.object({
  a: z.string().optional().default('hello'),
}).default({});

// Does not compile either, because `OptionalType` is not `Defaultable`.
// It should probably be when the wrapped type is.
const schema = z
  .object({
    a: z.string().default('hello').optional(),
  })
  .default({});

schema.parse(undefined)); // => { a: undefined } instead of the expected { a: 'hello' }
pierreis

comment created time in a month

Pull request review commentdavidmdm/myzod

Default values

 describe('Zod Parsing', () => {       //@ts-ignore       assert.ok(typeof schema.partial === 'undefined');     });++    it('should return default value if parsing undefined', () => {+      const schema = z.object({ a: z.string(), b: z.string() }).default({ a: 'hello', b: 'world' });+      assert.deepEqual(schema.parse(undefined), { a: 'hello', b: 'world' });+    });++    it('should return default value if parsing undefined - func', () => {+      const schema = z.object({ a: z.string(), b: z.string() }).default(() => ({ a: 'hello', b: 'world' }));+      assert.deepEqual(schema.parse(undefined), { a: 'hello', b: 'world' });+    });

I also wanted to create the following test case:

it('should return default value for nested values with default root', () => {
  const schema = z
    .object({
      a: z.string(),
      b: z.string().default(() => 'world').optional(),
    })
    .default({ a: 'hello' });
  assert.deepEqual(schema.parse(undefined), { a: 'hello', b: 'world' });
});

This does not work: the result is {a: "hello", b: undefined} instead of the expected value ; I am not yet sure why.

Also, as default() does not exist on OptionalType, z.string().default('world').optional() works while z.string().optional().default('world') fails type checking. Two things here:

– I think setting a default should imply optional, which is the whole point of having a default value in the first place.

default should work with optional values when the underlying type is Defaultable.

davidmdm

comment created time in a month

CommitCommentEvent

Pull request review commentdavidmdm/myzod

Default values

 describe('Zod Parsing', () => {       //@ts-ignore       assert.ok(typeof schema.partial === 'undefined');     });++    it('should return default value if parsing undefined', () => {+      const schema = z.object({ a: z.string(), b: z.string() }).default({ a: 'hello', b: 'world' });+      assert.deepEqual(schema.parse(undefined), { a: 'hello', b: 'world' });+    });++    it('should return default value if parsing undefined - func', () => {+      const schema = z.object({ a: z.string(), b: z.string() }).default(() => ({ a: 'hello', b: 'world' }));+      assert.deepEqual(schema.parse(undefined), { a: 'hello', b: 'world' });+    });

Can we add a test for nested values defaults? -- the following one is passing:

it('should return default value for nested values', () => {
    const schema = z.object({ a: z.string(), b: z.string().default("world") });
    assert.deepEqual(schema.parse({ a: "hello" }), { a: 'hello', b: 'world' });
});
davidmdm

comment created time in a month

pull request commentdavidmdm/myzod

Default values

That is awesome 🙂 This looks great — apart from a minor comment of mine.

davidmdm

comment created time in a month

CommitCommentEvent

issue commentdavidmdm/myzod

Default values

Nevermind that comment. I read your code a bit too quickly, it is all good!

pierreis

comment created time in a month

issue commentdavidmdm/myzod

Default values

No worries.

I would say that regarding infinite loops, it should probably be up to the developer to understand the possibility of this happening. I don't see it as a major issue.

Regarding the implementation, I have to admit I had no idea it was possible to perform a function call from a default function parameter. This looks good! 😄 Only the implementation of ObjectType.parse() will have to be modified I think ; as of now, it does not iterate on the fields of the shape.

No hurry here. I am not yet sure whether or not I will be able to tackle this before you do though.

pierreis

comment created time in a month

issue commentdavidmdm/myzod

Default values

Sounds good. The only edge case I can see are lazy types, but not quite sure about this. Regardless, I think that would be a great feature that would make my code significantly more straightforward.

Do you need help with this?

pierreis

comment created time in a month

issue openeddavidmdm/myzod

Default values

Hi @davidmdm -- would you be open to implement a method Type.default(T | () => T), that would set the default value on parse as the provided value if it is unknown?

That would be such that:

const userSchema = mz.Object({
    id: mz.string().default(uuidv4),
    name: mz.string(),
})
userSchema.parse({ name: "John }); // => { id: "2c5ea4c0-4067-11e9-8b2d-1b9d6bcdbbfd", name: "John" }

There are probably a few edge cases in implementing this, but I believe it would avoid a lot of boilerplate code when processing input API data.

created time in a month

starteddavidmdm/myzod

started time in a month

pull request commentdavidmdm/myzod

Return shape from ObjectType

Thanks!

pierreis

comment created time in a month

push eventpierreis/myzod

Pierre

commit sha 9fa0baafcb8191610f9909f28e0e4afb147f49b7

Update readme.md

view details

push time in a month

push eventpierreis/myzod

Pierre

commit sha 5547c8b84f0ed56cc028d79685be7be625c7849d

Update readme.md

view details

push time in a month

push eventpierreis/myzod

Pierre

commit sha 0fb1615b1f07f1d634df01ac1778e427b49bbc57

Add documentation for `Object.shape()`

view details

push time in a month

PR opened davidmdm/myzod

Return shape from ObjectType

This merge request adds a shape() method to ObjectType returning a copy of the type's ObjectShape. This eases the implementation of higher-level wrappers leveraging the object shape.

+4 -0

0 comment

1 changed file

pr created time in a month

push eventpierreis/myzod

Pierre

commit sha 99f4ff04a3330d9608014c0403d5a695b1d90a09

Return shape from ObjectType

view details

push time in a month

issue commentdavidmdm/myzod

Move predicate to the Type level

Got it, thanks for the update 🙂

pierreis

comment created time in a month

issue commentdavidmdm/myzod

Move predicate to the Type level

Instead of manually adding predicates to each types, why not consolidate this in a common abstract class from which all types accepting predicates would inherit?

pierreis

comment created time in a month

issue commentdavidmdm/myzod

Move predicate to the Type level

Looks good.

Made a comment about implementation simplification ; I don't think predicates should be either an array or a single predicate that can itself take multiple shapes. It makes the code harder to follow IMO. In addition, I think it hinders performance by additional unnecessary runtime checks and not leveraging JIT compilation as we should.

pierreis

comment created time in a month

CommitCommentEvent
CommitCommentEvent

issue commentdavidmdm/myzod

Move predicate to the Type level

Sounds good.

I think the predicates could be internal functions not exposed as part of the external API. The goal here is not necessarily to increase the API size, more to consolidate its internal implementation.

I am not suggesting to implement it this way, but technically, one could see the whole normalization process as a series of steps Array<(obj: unknown) => T | undefined that throws on error. This caters for everything, including type casting, type fitting and predicate processing.

pierreis

comment created time in a month

issue commentdavidmdm/myzod

Move predicate to the Type level

Sounds good. Two things I would add here:

First, the error message should be either a string or a function taking the object as parameter and returning a string. That would enable more descriptive error messages.

Second, in order to simplify the whole logic, I would actually suggest to switch the implementation of built-in functions min, max, ... as predicates. This would probably make the parser a tad simpler, and re-use the same logic for error processing in both cases.

The two would then be identical:

numberSchema.min(5, "number must be greater than 5")`
numberSchema.addPredicate(minValue(5), "number must be greater than 5")
pierreis

comment created time in a month

issue commentdavidmdm/myzod

Move predicate to the Type level

Hey -- I think that any non-trivial type could benefit from this. While I agree that it doesn't make much sense for boolean, undefined, null, enum or literal, it could make sense for virtually all the rest (numbers included):

– On numeric types, check that the number is odd, even, a power of two, ... – On object, check that complex field dependencies are satisfied (a password equals its confirmation, exactly one of [A, B or C keys are defined, etc] – On array, check that the number of elements is odd, even, ... – On date, check that it does fall on a weekday, ... – On a tuple, check that the elements are related in some way, ...

This is IMO so useful across the board that the cases for which it does not make sense are outnumbered by these for which it does. Which is why the request relates to moving it to the type level.

pierreis

comment created time in a month

issue openeddavidmdm/myzod

Add predicate at the Type level

Currently, predicates are handled at the String level. Is there any specific reason not to handle these at the Type level, so they can be used to validate any type?

created time in a month

issue commentvriad/zod

Support for custom validation

Sounds great. Do you need help in doing so?

pierreis

comment created time in a month

issue commentvriad/zod

Implement object

That second part is actually not the most complicated. It can be done roughly this way (written on the fly, untested, probably doesn't work)

interface ObjectShapeImpl<T extends ZodObject<any>> {
    assign: (data: Partial<T>) => T;
    with: (data: Partial<T> => T;
}

interface ObjectShapeStaticImpl<T extends ZodObject<any>> {
    new (obj: unknown): Infer<T> & ObjectShapeImpl<T>;
    // + shape(), type() or anything needed
}

Then, the class can be returned as:

function construct<T extends ZodObject<any>>(type: T) {
    return class StructClass<T> {
        private readonly value: Infer<T>;
        constructor(obj: Infer<T>) {
            for (const key of type.shape()) {
                Object.defineProperty(this, key, { ... });
            }
        )
    } as ObjectShapeImpl<T>
}
pierreis

comment created time in a month

issue openedvriad/zod

Implement object

Just like function implementation, it would be interesting to create an object implement(readonly: bool) method, that would create an anonymous class having the following functionality:

  • Has a constructor(t?: Type) that validates the object
  • Defines getters and setters (if not readonly)
  • May offer assign(other: Partial<Type>) or with(other: Partial<Type>), that respectively assign or copies the object and assign values, then runs validation

This would help creating ES6 classes inheriting from this base class, with validation built-in:

class MyUser extends userSchema.implement() {
    setPassword(pw: string) {
        this.password = hash(pw);
    }
}

There are a few unknowns for this: specifically, in case of nested objects that have an implementation, it would be great to return an instance of that implementation rather than a bare object. Some mapping between ZodType and implementation could theoretically be passed to Implement,

created time in a month

issue commentvriad/zod

Custom error messages

I think the typeError is good 👍 It could be combined with #37, to add a method typeError(ErrorMsg<Type>) to ZodType.

RyanMcDonald

comment created time in a month

issue commentvriad/zod

Support for custom validation

One way to perform this would be to add in a method in ZodType of the form:

validate(fn: Predicate<Type>, msg: ErrorMsg<Type>) => ZodType<Type, Def>

With:

type ErrorMsg<T> = string | ((obj: T) => string)
type Predicate<T> = (t: Type) => boolean
pierreis

comment created time in a month

starteddavidmdm/myzod

started time in a month

issue openedvriad/zod

Support for custom validation

Is there any plan to support custom validation? It could be of the form: z.string().validate(someValidationFunction).

That would be a great help to build advanced schemas.

This could also take the form of custom types, but for now, z.ZodType / z.ZodTypeDef not being exposed as part of the API make it a hack IMO.

created time in a month

issue closedhanwen/go-fuse

Investigating a possible memory leak in lookup

I am currently investigating a possible memory leak in the rawBridge.Lookup method, causing the following allocation to appear in the profile (in my lookup implementation):

   20.90GB    20.90GB     83:   inode := fsNode{
         .          .     84:           dataID:       dataID,
         .          .     85:           relativePath: relativePath,
         .          .     86:           dataPath:     dataPath,
         .          .     87:           root:         n.root,
         .          .     88:   }
         .    10.61GB     89:   child := n.NewInode(ctx, &inode, stableAttrs)
         .          .     90:
         .          .     91:   return child, 0

Note that nothing is done for caching these nodes, and they are returned straight to the underlying go-fuse implementation.

The heap trace is the following:

Heap trace

My understanding is that of the approximately 30 GB generated from the Lookup method, 20 related to the fsNode struct, that is referenced from the Inode, and not garbage collected until the latter goes out of scope. I cannot figure out where the Inode could be retained.

The file system is mounted with the following options:

AttrTimeout: &time.Second,
EntryTimeout: &time.Second,
NegativeTimeout: nil,
MountOptions: {
  DisableXAttrs: true,
  IgnoreSecurityLabels: true,
  AllowOther: true,
  RememberInodes: false,
}

It looks like go-fuse is retaining this memory in addNewChild:

   31.85GB    31.85GB    120:         parent.setEntry(name, child)

Hence the question: I am trying to build a dynamically discoverable file system aggregating large numbers of inodes. I am not sure how to update the behavior of the library so that the tree is not kept in memory.

Looking at the code, it is not clear to me what the mechanism to evict inodes is -- it seems that as long as these root nodes are remembered somewhere, all their hierarchy will also be forever.

closed time in 2 months

pierreis

issue commenthanwen/go-fuse

Investigating a possible memory leak in lookup

After extensive testing, it appears that the difference between basil.org/fuse and go-fuse boils down to executing on servers with a different setting for vfs_cache_pressure, which did cause the server running basil.org/fuse to forget nodes significantly more aggressively. When deployed with the same setting of 100, the memory usage of both tends to increase due to the kernel tendency to retain cache.

Adding active notification to forget cache items helped keeping the memory usage stable over multiple hours. The test case you developed helped a ton in debugging this issue.

Appreciate the help immensely.

Closing this issue.

pierreis

comment created time in 2 months

issue closedhanwen/go-fuse

Panic on unknown node after NotifyEntry

I regularly get the following panic from go-fuse:

2020/04/08 16:38:53 unknown node 10048884762121421891
panic: unknown node 10048884762121421891

goroutine 823 [running]:
log.Panicf(0xb418f5, 0xf, 0xc000d0fdd0, 0x1, 0x1)
        GOROOT/src/log/log.go:358 +0xc0
github.com/hanwen/go-fuse/v2/fs.(*rawBridge).inode(0xc00039a000, 0x8b74cf732b154443, 0x0, 0x0, 0x0)
        external/com_github_hanwen_go_fuse_v2/fs/bridge.go:227 +0x17b
github.com/hanwen/go-fuse/v2/fs.(*rawBridge).OpenDir(0xc00039a000, 0xc0026ac840, 0xc000f58198, 0xc000f58108, 0xc000000000)
        external/com_github_hanwen_go_fuse_v2/fs/bridge.go:772 +0x72
github.com/hanwen/go-fuse/v2/fuse.doOpenDir(0xc00030fce0, 0xc000f58000)
        external/com_github_hanwen_go_fuse_v2/fuse/opcode.go:182 +0x74
github.com/hanwen/go-fuse/v2/fuse.(*Server).handleRequest(0xc00030fce0, 0xc000f58000, 0xc000000000)
        external/com_github_hanwen_go_fuse_v2/fuse/server.go:470 +0x3b7
github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc00030fce0, 0xc001353501)
        external/com_github_hanwen_go_fuse_v2/fuse/server.go:437 +0x18a
created by github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest
        external/com_github_hanwen_go_fuse_v2/fuse/server.go:304 +0x417

This started to happen after adding a LRU cache in my application, actively sending NotifyEntry when an Inode is evicted from the cache. I assume that evicting a parent folder of a node referenced by a subsequent request may have side effect, but these do not seem to be documented.

Should this happen (I am not sure whether it derives from a normal, expected condition or from a bug in the stack), wouldn't it be more appropriate to respond an error instead of panicking?

This may relate to #334.

closed time in 2 months

pierreis

issue commenthanwen/go-fuse

Panic on unknown node after NotifyEntry

Thanks Han-Wen. That sounds great. Indeed, I cannot reproduce the issue on master. Thank you for your time.

Closing the issue.

pierreis

comment created time in 2 months

issue commenthanwen/go-fuse

Panic on unknown node after NotifyEntry

Interesting. Let me try with master and update this report accordingly.

pierreis

comment created time in 2 months

issue commenthanwen/go-fuse

Panic on unknown node after NotifyEntry

Release 2.0.2. Built with Bazel, not sure whether it does some magic tricks with the go files.

pierreis

comment created time in 2 months

issue commenthanwen/go-fuse

Panic on unknown node after NotifyEntry

Using automatic inodes seems to prevent the issue from occurring as well.

pierreis

comment created time in 2 months

issue commenthanwen/go-fuse

Panic on unknown node after NotifyEntry

An extract of the debug log seems to confirm the theory:

2020/04/09 10:23:26 rx 72017: GETATTR i11972922661925671148 {Fh 4}
2020/04/09 10:23:26 tx 72017:     OK, {tA=1s {M0100644 SZ=199902 L=1 0:0 B0*0 i0:11972922661925671148 A 1586382326.000000 M 1586382326.000000 C 1586382326.000000}}
2020/04/09 10:23:26 rx 72018: FLUSH i11972922661925671148 {Fh 4}
2020/04/09 10:23:26 tx 72018:     OK
2020/04/09 10:23:26 rx 72019: RELEASE i11972922661925671148 {Fh 4 0x8000  L0}
2020/04/09 10:23:26 rx 72021: RELEASEDIR i408811860997008763 {Fh 3 NONBLOCK,DIRECTORY,0x28000  L0}
2020/04/09 10:23:26 tx 72019:     OK
2020/04/09 10:23:26 rx 72020: RELEASEDIR i1489782920959277745 {Fh 6 NONBLOCK,DIRECTORY,0x28000  L0}
2020/04/09 10:23:26 tx 72021:     OK
2020/04/09 10:23:26 tx 72020:     OK
2020/04/09 10:23:26 rx 72023: LOOKUP i8218206828995173969 ["TEST"] 21b
2020/04/09 10:23:26 rx 72022: RELEASEDIR i5382203406692611568 {Fh 2 NONBLOCK,DIRECTORY,0x28000  L0}
2020/04/09 10:23:26 tx 72022:     OK
2020/04/09 10:23:26 tx 0:     NOTIFY_INVAL_ENTRY, {parent i8218206828995173969 sz 20} "TEST"
2020/04/09 10:23:26 Response: ENTRY_NOTIFY: OK
2020/04/09 10:23:26 tx 72023:     OK, {i10048884762121421891 g0 tE=1s tA=1s {M040755 SZ=4096 L=1 0:0 B0*0 i0:10048884762121421891 A 1577810722.000000 M 1577810722.000000 C 1577810722.000000}}
2020/04/09 10:23:26 rx 72024: FORGET i10048884762121421891 {Nlookup=2}
2020/04/09 10:23:26 rx 72025: LOOKUP i8218206828995173969 ["TEST"] 21b
2020/04/09 10:23:26 tx 72025:     OK, {i10048884762121421891 g0 tE=1s tA=1s {M040755 SZ=4096 L=1 0:0 B0*0 i0:10048884762121421891 A 1577810722.000000 M 1577810722.000000 C 1577810722.000000}}
2020/04/09 10:23:26 rx 72026: OPENDIR i10048884762121421891
2020/04/09 10:23:26 unknown node 10048884762121421891
panic: unknown node 10048884762121421891

goroutine 446 [running]:
log.Panicf(0xb418f5, 0xf, 0xc001f45dd0, 0x1, 0x1)
        GOROOT/src/log/log.go:358 +0xc0
github.com/hanwen/go-fuse/v2/fs.(*rawBridge).inode(0xc000190500, 0x8b74cf732b154443, 0x0, 0x0, 0x0)
        external/com_github_hanwen_go_fuse_v2/fs/bridge.go:227 +0x17b
github.com/hanwen/go-fuse/v2/fs.(*rawBridge).OpenDir(0xc000190500, 0xc0011fc000, 0xc00085d818, 0xc00085d788, 0x0)
        external/com_github_hanwen_go_fuse_v2/fs/bridge.go:772 +0x72
github.com/hanwen/go-fuse/v2/fuse.doOpenDir(0xc0001ad4a0, 0xc00085d680)
        external/com_github_hanwen_go_fuse_v2/fuse/opcode.go:182 +0x74
github.com/hanwen/go-fuse/v2/fuse.(*Server).handleRequest(0xc0001ad4a0, 0xc00085d680, 0xc000000000)
        external/com_github_hanwen_go_fuse_v2/fuse/server.go:470 +0x3b7
github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc0001ad4a0, 0xc001f6f001)
        external/com_github_hanwen_go_fuse_v2/fuse/server.go:437 +0x18a
created by github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest
        external/com_github_hanwen_go_fuse_v2/fuse/server.go:304 +0x417

According to this, go-fuse is still processing an opendir request after the forget has been received.

pierreis

comment created time in 2 months

issue commenthanwen/go-fuse

Panic on unknown node after NotifyEntry

If this is the case, it would be interesting to understand which race condition is at work here. My workload makes go-fuse panic within a minute or so with multiple goroutines, but has been working fine for a few hours in single thread mode.

pierreis

comment created time in 2 months

issue commenthanwen/go-fuse

Panic on unknown node after NotifyEntry

This looks a lot like a race condition to me. The problem doesn't seem to be reproduced when go-fuse is running in single-threaded mode. Hence, one theory:

Although the kernel guarantees that a node will not be referenced after a FORGET message has been sent, it may happen that a forgotten node is referenced by a previous request in the queue, handled by a goroutine scheduled by the runtime to run after the forget.

Would this make sense?

pierreis

comment created time in 2 months

issue openedhanwen/go-fuse

Panic on unknown node after NotifyEntry

I regularly get the following panic from go-fuse:

2020/04/08 16:38:53 unknown node 10048884762121421891 panic: unknown node 10048884762121421891

goroutine 823 [running]: log.Panicf(0xb418f5, 0xf, 0xc000d0fdd0, 0x1, 0x1) GOROOT/src/log/log.go:358 +0xc0 github.com/hanwen/go-fuse/v2/fs.(*rawBridge).inode(0xc00039a000, 0x8b74cf732b154443, 0x0, 0x0, 0x0) external/com_github_hanwen_go_fuse_v2/fs/bridge.go:227 +0x17b github.com/hanwen/go-fuse/v2/fs.(*rawBridge).OpenDir(0xc00039a000, 0xc0026ac840, 0xc000f58198, 0xc000f58108, 0xc000000000) external/com_github_hanwen_go_fuse_v2/fs/bridge.go:772 +0x72 github.com/hanwen/go-fuse/v2/fuse.doOpenDir(0xc00030fce0, 0xc000f58000) external/com_github_hanwen_go_fuse_v2/fuse/opcode.go:182 +0x74 github.com/hanwen/go-fuse/v2/fuse.(*Server).handleRequest(0xc00030fce0, 0xc000f58000, 0xc000000000) external/com_github_hanwen_go_fuse_v2/fuse/server.go:470 +0x3b7 github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc00030fce0, 0xc001353501) external/com_github_hanwen_go_fuse_v2/fuse/server.go:437 +0x18a created by github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest external/com_github_hanwen_go_fuse_v2/fuse/server.go:304 +0x417

This started to happen after adding a LRU cache in my application, actively sending NotifyEntry when an Inode is evicted from the cache. I assume that evicting a parent folder of a node referenced by a subsequent request may have side effect, but these do not seem to be documented.

Should this happen (I am not sure whether it derives from a normal, expected condition or from a bug in the stack), wouldn't it be more appropriate to respond an error instead of panicking?

This may relate to #334.

created time in 2 months

issue commenthanwen/go-fuse

Investigating a possible memory leak in lookup

I am trying various ways to understand what is happening here. I am starting to suspect that the leak-like appearance could be due to the way the go runtime hooks to its allocated memory combined with the kernel being lazy forgetting nodes.

The memory increase with Brazil's FUSE is much slower (probably due to a significantly lower memory size per inode), but nonetheless growing after a few days under sustained load.

I am currently integrating active eviction from the application using an LRU cache to try mitigate the issue (which is, AFAIK, the way it is handled in Ceph). I'll update as soon as I get more details on this.

pierreis

comment created time in 2 months

issue commenthanwen/go-fuse

Investigating a possible memory leak in lookup

Sounds great, I’ll do this and let you know.

pierreis

comment created time in 2 months

push eventtyr/fuse

Pierre

commit sha edf385ef24f0905a76db7c4f235ce8c9d30545a2

Fix build

view details

push time in 2 months

push eventtyr/fuse

Pierre

commit sha 0123593da1fd3c7f6a6822627203c810469dfbfd

Set request processors according to the box CPUs

view details

push time in 2 months

issue commenthanwen/go-fuse

Investigating a possible memory leak in lookup

Thanks. I'll continue digging on my side. Please advise how I can assist you best.

pierreis

comment created time in 2 months

issue commenthanwen/go-fuse

Investigating a possible memory leak in lookup

To be sure that the issue is related to go-fuse and not to my code, I did adjust the few methods I am implementing to use Bazil's FUSE library. I see no such memory issue, and the memory is stable over time, with the same workload. It would tend to indicate that there is indeed some memory leak in go-fuse preventing it from dereferencing nodes despite FORGET requests being received by the kernel, but I still cannot pinpoint where.

pierreis

comment created time in 2 months

push eventtyr/fuse

Pierre

commit sha 62947f31423153dce7770789318afaa5a49e2dcd

Limit number of workers

view details

push time in 2 months

issue commenthanwen/go-fuse

Investigating a possible memory leak in lookup

I see forget coming from the kernel regularly, also when clearing the cache manually. My issue is that inodes do not seem to be garbage collected despite these requests.

pierreis

comment created time in 2 months

issue commenthanwen/go-fuse

Investigating a possible memory leak in lookup

I just tried, it doesn't seem to have any impact at all on the running process unfortunately.

pierreis

comment created time in 2 months

issue openedhanwen/go-fuse

Investigating a possible memory leak in lookup

I am currently investigating a possible memory leak in the rawBridge.Lookup method, causing the following allocation to appear in the profile (in my lookup implementation):

   20.90GB    20.90GB     83:   inode := fsNode{
         .          .     84:           dataID:       dataID,
         .          .     85:           relativePath: relativePath,
         .          .     86:           dataPath:     dataPath,
         .          .     87:           root:         n.root,
         .          .     88:   }
         .    10.61GB     89:   child := n.NewInode(ctx, &inode, stableAttrs)
         .          .     90:
         .          .     91:   return child, 0

Note that nothing is done for caching these nodes, and they are returned straight to the underlying go-fuse implementation.

The heap trace is the following:

Heap trace

created time in 2 months

more