profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/neil-gok/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

issue commentgoogleapis/google-cloud-dotnet

Firestore - Limit not returning correct number of records

I appreciate you looking into this.

neil-gok

comment created time in 2 days

issue commentgoogleapis/google-cloud-dotnet

Firestore - Limit not returning correct number of records

@jskeet

One more question on this - we worked with Google a few months ago to turn on the Experimental Long Polling option in the Firestore Console, due to some proxy restrictions on our side that were preventing documents from loading. Is there a way to enable that as well with the .net Firestore Library, for when we are debugging on our corporate network? This could alleviate some of the anomalies that we see, such as the one we saw here.

neil-gok

comment created time in 2 days

issue closedgoogleapis/google-cloud-dotnet

Firestore - Limit not returning correct number of records

Library: Google.Cloud.Firestore 2.3.1

When attempting to query a range of dates, I am unable to return the correct number of records when including a limit in the query. In the example below, without the limit selected, it returns over 5k records. When I apply a limit of 5k, it only returns 2256 records.

        public async Task<List<T>> QueryRecords<T>(Query query) where T : FireStoreBase
        {
            QuerySnapshot querySnapshot = null;
            querySnapshot = await query.GetSnapshotAsync();            
            List<T> list = new List<T>();
            var count = querySnapshot.Documents.Count;
            foreach (DocumentSnapshot documentSnapshot in querySnapshot.Documents)
            {
                if (documentSnapshot.Exists)
                {                    
                    Dictionary<string, object> record = documentSnapshot.ToDictionary();                    
                    T newItem = documentSnapshot.ConvertTo<T>();
                    newItem.Id = documentSnapshot.Id;
                    list.Add(newItem);
                }
            }
            return list;
        }

        public async Task<List<SomeData>> GetSchedulesFilterByDate(DateTime startDate, DateTime endDate, int recordsLimit = 0)
        {
            if (recordsLimit == 0) recordsLimit = _recordsLimit;

            Query query = repo.fireStoreDb.Collection(collectionName).OrderBy("SomeDate")
                                            .WhereGreaterThanOrEqualTo("SomeDate", startDate)
                                            .WhereLessThanOrEqualTo("SomeDate", endDate).Limit(5000);

            return await repo.QueryRecords<SomeData>(query);
        }

closed time in 3 days

neil-gok

issue commentgoogleapis/google-cloud-dotnet

Firestore - Limit not returning correct number of records

@jskeet

Thanks for your assistance with this. After trying a number of things on my side, I have determined that this is a proxy issue, and not an issue with the Library or the Firestore API itself.

neil-gok

comment created time in 3 days

issue commentgoogleapis/google-cloud-dotnet

Firestore - Limit not returning correct number of records

@jskeet

I am unable to repro when creating a new collection with mock data, which leads me to believe there is either something in the data itself, or the number of documents in the production collection that is causing this issue. We have hundreds of thousands of records in our Prod collection, while my test collection only has a few thousand.

I will continue attempting to repro, once I have seeded the collection with a similar number of documents to our prod collection.

Neil

neil-gok

comment created time in 7 days

issue commentgoogleapis/google-cloud-dotnet

Firestore - Limit not returning correct number of records

@jskeet,

My apologies. I will create script to seed a collection, and also include the model I’m working with. I will update my response once I have something put together.

neil-gok

comment created time in 8 days

issue commentgoogleapis/google-cloud-dotnet

Firestore - Limit not returning correct number of records

Jon,

I wonder if you use a more complex model, if you would see the same results. My model has nested objects, which might make the difference.

Neil

neil-gok

comment created time in 8 days

issue commentgoogleapis/google-cloud-dotnet

Firestore - Limit not returning correct number of records

Jon,

I created a similar example as yours.

`static async Task Main(string[] args) { try {

            var json = File.ReadAllText(Environment.GetEnvironmentVariable("GOOGLE_APPLICATION_CREDENTIALS"));

            var cr = (Newtonsoft.Json.Linq.JObject)JsonConvert.DeserializeObject(json);

            var projectId = (string)cr.GetValue("project_id");

            string collectionId = "Schedules";

            DateTime start = DateTime.ParseExact(
            args[0], "yyyy-MM-dd'T'HH:mm", CultureInfo.InvariantCulture,
            DateTimeStyles.AdjustToUniversal | DateTimeStyles.AssumeUniversal);
            DateTime end = DateTime.ParseExact(
                args[1], "yyyy-MM-dd'T'HH:mm", CultureInfo.InvariantCulture,
                DateTimeStyles.AdjustToUniversal | DateTimeStyles.AssumeUniversal);
            
            int? limit = args.Length > 2 ? int.Parse(args[2]) : default(int?);

            FirestoreDb db = FirestoreDb.Create(projectId);
            
            var collection = db.Collection(collectionId);

            var query = collection
                .OrderBy("InterviewDate")
                .WhereGreaterThanOrEqualTo("InterviewDate", start)
                .WhereLessThan("InterviewDate", end);

            if (limit is int actualLimit)
            {
                query = query.Limit(actualLimit);
            }

            var snapshot = await query.GetSnapshotAsync();

            Console.WriteLine($"Query results: {snapshot.Count}");
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
        }
    }`

My results

` $ dotnet run -- 2021-10-07T00:00 2021-10-13T00:00 Query results: 9602

$ dotnet run -- 2021-10-07T00:00 2021-10-13T00:00 5000 Query results: 2243 `

neil-gok

comment created time in 8 days

issue commentgoogleapis/google-cloud-dotnet

Firestore - Limit not returning correct number of records

Jon,

Responses to your questions below.

What does querySnapshot.Documents.Count return? 2257 Could you log any documents that don't meet your condition of documentSnapshot.Exists? There are no documents that do not meet my condition.

I'll work on providing a complete example tomorrow.

Neil

neil-gok

comment created time in 10 days

issue openedgoogleapis/google-cloud-dotnet

Firestore - Limit not returning correct number of records

Library: Google.Cloud.Firestore 2.3.1

When attempting to query a range of dates, I am unable to return the correct number of records when including a limit in the query. In the example below, without the limit selected, it returns over 5k records. When I apply a limit of 5k, it only returns 2256 records.

public async Task<List<T>> QueryRecords<T>(Query query) where T : FireStoreBase { QuerySnapshot querySnapshot = null; querySnapshot = await query.GetSnapshotAsync();
List<T> list = new List<T>(); var count = querySnapshot.Documents.Count; foreach (DocumentSnapshot documentSnapshot in querySnapshot.Documents) { if (documentSnapshot.Exists) {
Dictionary<string, object> record = documentSnapshot.ToDictionary();
T newItem = documentSnapshot.ConvertTo<T>(); newItem.Id = documentSnapshot.Id; list.Add(newItem); } } return list; }

 public async Task<List<SomeData>> GetSchedulesFilterByDate(DateTime startDate, DateTime endDate, int recordsLimit = 0)
    {
        if (recordsLimit == 0) recordsLimit = _recordsLimit;

        Query query = repo.fireStoreDb.Collection(collectionName).OrderBy("SomeDate")
                                        .WhereGreaterThanOrEqualTo("SomeDate", startDate)
                                        .WhereLessThanOrEqualTo("SomeDate", endDate).Limit(5000);

        return await repo.QueryRecords<SomeData>(query);
    }

created time in 13 days

issue commentwebpack/webpack-dev-server

ReDoS Vulnerability

A ticket has been open for a while, but ansi-html is not being maintained. I'm asking if webpack will continue to use an unmaintained library which has a known vulnerability, or are there plans to replace it?

neil-gok

comment created time in 3 months

issue openedwebpack/webpack-dev-server

ReDoS Vulnerability

  • [x] This is a bug
  • [ ] This is a modification request

For Bugs; How can we reproduce the behavior?

This is a vulnerability in ansi-html, which this package depends on. To repro, you can use the command below as described https://github.com/Tjatse/ansi-html/issues/19.

require('ansi-html')('\x1b[0m\x1b[' + '0'.repeat(35))

Are there any plans to replace/remove this package, with a more active project? The library maintainers have been made aware of this change, but it does not look like it is being maintained. This is preventing us from moving our source code to Production, as ansi-html is being flagged.

created time in 3 months