profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/chandresh-pancholi/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Chandresh Pancholi chandresh-pancholi Bangalore Contributor

chandresh-pancholi/developer-to-engineer 2

👷 A reading list to help transform you from a developer to an engineer

chandresh-pancholi/engineering-blogs 1

A curated list of engineering blogs

chandresh-pancholi/argo 0

Container-native workflows for Kubernetes.

chandresh-pancholi/awesome-go 0

A curated list of awesome Go frameworks, libraries and software

chandresh-pancholi/awesome-interview-questions 0

:octocat: A curated awesome list of lists of interview questions. Feel free to contribute! :mortar_board:

chandresh-pancholi/awesome-java 0

A curated list of awesome Java frameworks, libraries and software.

chandresh-pancholi/awesome-mongodb 0

:leaves: A curated list of awesome MongoDB resources, libraries, tools and applications

chandresh-pancholi/awesome-prometheus 0

A curated list of awesome Prometheus resources, projects and tools.

chandresh-pancholi/awesome-react-native 0

Awesome React Native components, news, tools, and learning material!

chandresh-pancholi/Awesome-React-Native-Education 0

Use this to learn React Native

release layer5io/meshery

v0.5.24

released time in a day

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public Props fetchProjectProperty(Project project, String propsName)     }   } +  public Set<Integer> fetchProjectIdsByEventType(EventType eventType) throws ProjectManagerException {+    QueryRunner runner = createQueryRunner();

Don't we need connection here like you did in fetchAllInactiveProjects?

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 private void cleanOlderProjectVersion(Connection connection, int projectId,     }   } +  private static class ProjectIdsByEventTypeResultHandler implements+          ResultSetHandler<Set<Integer>> {+    private static final String SELECT_PROJECTS_BY_EVENT_TYPE =+            "SELECT distinct(project_id) from project_events WHERE `event_type`=?";

This will be costly as it will scan significant data. Should we keep this in the main sync block ?

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";++    private boolean shutdown = false;+    private long lastProjectFileCleanTime = -1;+    private long lastProjectCleanTime = -1;++    public CleanerThread() {+      this.setName("ProjectManager-Cleaner-Thread");+      logger.info("Starting cleaner thread: " + this.getName());+    }++    @SuppressWarnings("unused")+    public void shutdown() {

Who is calling this ? Worst case we need to register a shutdownhook isn't it ?

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public JdbcProjectLoader(Props props) {     return projects;   } +  private List<Project> fetchAllInactiveProjects(Connection connection)+          throws ProjectManagerException {+    QueryRunner runner = new QueryRunner();++    ProjectResultHandler handler = new ProjectResultHandler();+    List<Project> projects = null;+    try {+      projects =+              runner.query(connection,+                      ProjectResultHandler.SELECT_ALL_INACTIVE_PROJECTS, handler);++      for (Project project : projects) {

Common code w.r.t to fetchAllActiveProjects, can we avoid it

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public JdbcProjectLoader(Props props) {     return projects;   } +  @Override+  public List<Project> fetchAllInactiveProjects() throws ProjectManagerException {+    Connection connection = getConnection();++    List<Project> projects = null;+    try {+      projects = fetchAllInactiveProjects(connection);

simply return here w/p projects variable ?

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";++    private boolean shutdown = false;+    private long lastProjectFileCleanTime = -1;+    private long lastProjectCleanTime = -1;++    public CleanerThread() {+      this.setName("ProjectManager-Cleaner-Thread");+      logger.info("Starting cleaner thread: " + this.getName());+    }++    @SuppressWarnings("unused")+    public void shutdown() {+      shutdown = true;+      this.interrupt();+    }++    public void run() {+      while (!shutdown) {+        synchronized (this) {

Did we use sync only for the wait() below? Can't we rather use sleep(ms) - this won't need sync ? My issue or doubt here is, the functionality being done is a bit heavy to be done under a single lock. Either remove sync or keep only necessary critical section with sync

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public void cleanOlderProjectVersion(int projectId, int version)    } +  @Override+  public void cleanProjectFiles(Project project) throws ProjectManagerException {+    // TODO Auto-generated method stub

UTs missing ?

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";++    private boolean shutdown = false;+    private long lastProjectFileCleanTime = -1;+    private long lastProjectCleanTime = -1;++    public CleanerThread() {+      this.setName("ProjectManager-Cleaner-Thread");+      logger.info("Starting cleaner thread: " + this.getName());+    }++    @SuppressWarnings("unused")+    public void shutdown() {+      shutdown = true;+      this.interrupt();+    }++    public void run() {+      while (!shutdown) {+        synchronized (this) {+          try {+            long currentTime = System.currentTimeMillis();++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+              cleanProjectFilesAndRemoveProject();+              lastProjectFileCleanTime = currentTime;+            }++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+              purgeRemovedProjects();+              lastProjectCleanTime = currentTime;+            }++            logger.info("Cleaner Thread completed. Waiting!!!!!");+            wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+          } catch (ProjectManagerException e) {+            logger.error(this.getName() + "failed due to: " + e.getMessage());+          } catch (InterruptedException e) {+            logger.info("Interrupted. Probably to shut down.");+          }+        }+      }+    }++    private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+      User deleter = new User(DELETER);+      long currentTime = System.currentTimeMillis();++      Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+      logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++      List<Project> projects = projectsById.values()+              .stream()+              .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+                      currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+              .collect(Collectors.toList());++      logger.info("Project files to clean: " + projects.size());++      for(Project project : projects) {+        logger.debug("Cleaning project files for project: " + project.getId());

log info perhaps as we are cleaning up ?

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";++    private boolean shutdown = false;+    private long lastProjectFileCleanTime = -1;+    private long lastProjectCleanTime = -1;++    public CleanerThread() {+      this.setName("ProjectManager-Cleaner-Thread");+      logger.info("Starting cleaner thread: " + this.getName());+    }++    @SuppressWarnings("unused")+    public void shutdown() {+      shutdown = true;+      this.interrupt();+    }++    public void run() {+      while (!shutdown) {+        synchronized (this) {+          try {+            long currentTime = System.currentTimeMillis();++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+              cleanProjectFilesAndRemoveProject();+              lastProjectFileCleanTime = currentTime;+            }++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+              purgeRemovedProjects();+              lastProjectCleanTime = currentTime;+            }++            logger.info("Cleaner Thread completed. Waiting!!!!!");+            wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+          } catch (ProjectManagerException e) {+            logger.error(this.getName() + "failed due to: " + e.getMessage());+          } catch (InterruptedException e) {+            logger.info("Interrupted. Probably to shut down.");+          }+        }+      }+    }++    private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+      User deleter = new User(DELETER);+      long currentTime = System.currentTimeMillis();++      Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+      logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++      List<Project> projects = projectsById.values()+              .stream()+              .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+                      currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+              .collect(Collectors.toList());++      logger.info("Project files to clean: " + projects.size());

can we log only if size is > 0

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";

Instead can't we simply have a private static final User deleter = new User(DELETER); ?

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";++    private boolean shutdown = false;+    private long lastProjectFileCleanTime = -1;+    private long lastProjectCleanTime = -1;++    public CleanerThread() {+      this.setName("ProjectManager-Cleaner-Thread");+      logger.info("Starting cleaner thread: " + this.getName());+    }++    @SuppressWarnings("unused")+    public void shutdown() {+      shutdown = true;+      this.interrupt();+    }++    public void run() {+      while (!shutdown) {+        synchronized (this) {+          try {+            long currentTime = System.currentTimeMillis();++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+              cleanProjectFilesAndRemoveProject();+              lastProjectFileCleanTime = currentTime;+            }++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+              purgeRemovedProjects();+              lastProjectCleanTime = currentTime;+            }++            logger.info("Cleaner Thread completed. Waiting!!!!!");+            wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+          } catch (ProjectManagerException e) {+            logger.error(this.getName() + "failed due to: " + e.getMessage());+          } catch (InterruptedException e) {+            logger.info("Interrupted. Probably to shut down.");+          }+        }+      }+    }++    private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+      User deleter = new User(DELETER);+      long currentTime = System.currentTimeMillis();++      Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+      logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++      List<Project> projects = projectsById.values()+              .stream()+              .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+                      currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+              .collect(Collectors.toList());++      logger.info("Project files to clean: " + projects.size());++      for(Project project : projects) {+        logger.debug("Cleaning project files for project: " + project.getId());+        projectLoader.cleanProjectFiles(project);+        logger.debug("Removing project: " + project.getId());+        removeProject(project, deleter);+      }+    }++    private void purgeRemovedProjects() throws ProjectManagerException {+      long currentTime = System.currentTimeMillis();++      List<Project> inactiveProjects = projectLoader.fetchAllInactiveProjects();+      logger.info("Inactive projects count: " + inactiveProjects.size());++      Set<Integer> purgedProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.PURGE);+      logger.info("Purged projects count: " + purgedProjectIds.size());++      List<Project> projectsToPurge = inactiveProjects+              .stream()+              .filter(project -> !purgedProjectIds.contains(project.getId()) &&+                      currentTime - project.getLastModifiedTimestamp() > removedProjectRetentionMs)+              .collect(Collectors.toList());++      logger.info("Projects to purge count: " + projectsToPurge.size());++      User deleter = new User(DELETER);+      for(Project project: projectsToPurge) {+        logger.debug("Purging project: " + project.getId());+        purgeProject(project, deleter);+      }+    }   }    private void loadAllProjects() {     List<Project> projects;     try {       projects = projectLoader.fetchAllActiveProjects();+      logger.info("Active projects count: " + projects.size());

Do we need this? We have enough logging above

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";++    private boolean shutdown = false;+    private long lastProjectFileCleanTime = -1;+    private long lastProjectCleanTime = -1;

lastProjectCleanTime -> lastDeactivatedProjectCleanTime ?

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";++    private boolean shutdown = false;+    private long lastProjectFileCleanTime = -1;+    private long lastProjectCleanTime = -1;++    public CleanerThread() {+      this.setName("ProjectManager-Cleaner-Thread");+      logger.info("Starting cleaner thread: " + this.getName());+    }++    @SuppressWarnings("unused")+    public void shutdown() {+      shutdown = true;+      this.interrupt();+    }++    public void run() {+      while (!shutdown) {+        synchronized (this) {+          try {+            long currentTime = System.currentTimeMillis();++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+              cleanProjectFilesAndRemoveProject();+              lastProjectFileCleanTime = currentTime;+            }++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+              purgeRemovedProjects();+              lastProjectCleanTime = currentTime;+            }++            logger.info("Cleaner Thread completed. Waiting!!!!!");+            wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+          } catch (ProjectManagerException e) {+            logger.error(this.getName() + "failed due to: " + e.getMessage());+          } catch (InterruptedException e) {+            logger.info("Interrupted. Probably to shut down.");+          }+        }+      }+    }++    private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+      User deleter = new User(DELETER);+      long currentTime = System.currentTimeMillis();++      Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+      logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++      List<Project> projects = projectsById.values()+              .stream()+              .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+                      currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+              .collect(Collectors.toList());++      logger.info("Project files to clean: " + projects.size());++      for(Project project : projects) {+        logger.debug("Cleaning project files for project: " + project.getId());+        projectLoader.cleanProjectFiles(project);+        logger.debug("Removing project: " + project.getId());+        removeProject(project, deleter);+      }+    }++    private void purgeRemovedProjects() throws ProjectManagerException {+      long currentTime = System.currentTimeMillis();++      List<Project> inactiveProjects = projectLoader.fetchAllInactiveProjects();+      logger.info("Inactive projects count: " + inactiveProjects.size());++      Set<Integer> purgedProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.PURGE);+      logger.info("Purged projects count: " + purgedProjectIds.size());++      List<Project> projectsToPurge = inactiveProjects+              .stream()+              .filter(project -> !purgedProjectIds.contains(project.getId()) &&+                      currentTime - project.getLastModifiedTimestamp() > removedProjectRetentionMs)+              .collect(Collectors.toList());++      logger.info("Projects to purge count: " + projectsToPurge.size());

can you club these 3 logs into 1 and that too is projectsToPurge.size() > 0 to avoid spamming logs

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;

Shouldn't this be configurable as well ?

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";++    private boolean shutdown = false;+    private long lastProjectFileCleanTime = -1;+    private long lastProjectCleanTime = -1;++    public CleanerThread() {+      this.setName("ProjectManager-Cleaner-Thread");+      logger.info("Starting cleaner thread: " + this.getName());+    }++    @SuppressWarnings("unused")+    public void shutdown() {+      shutdown = true;+      this.interrupt();+    }++    public void run() {+      while (!shutdown) {+        synchronized (this) {+          try {+            long currentTime = System.currentTimeMillis();++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+              cleanProjectFilesAndRemoveProject();+              lastProjectFileCleanTime = currentTime;+            }++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+              purgeRemovedProjects();+              lastProjectCleanTime = currentTime;+            }++            logger.info("Cleaner Thread completed. Waiting!!!!!");+            wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+          } catch (ProjectManagerException e) {+            logger.error(this.getName() + "failed due to: " + e.getMessage());+          } catch (InterruptedException e) {+            logger.info("Interrupted. Probably to shut down.");+          }+        }+      }+    }++    private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+      User deleter = new User(DELETER);+      long currentTime = System.currentTimeMillis();++      Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+      logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++      List<Project> projects = projectsById.values()+              .stream()+              .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+                      currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+              .collect(Collectors.toList());++      logger.info("Project files to clean: " + projects.size());

info(Project files to clean for: " + size + " projects")

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";++    private boolean shutdown = false;+    private long lastProjectFileCleanTime = -1;+    private long lastProjectCleanTime = -1;++    public CleanerThread() {+      this.setName("ProjectManager-Cleaner-Thread");+      logger.info("Starting cleaner thread: " + this.getName());+    }++    @SuppressWarnings("unused")+    public void shutdown() {+      shutdown = true;+      this.interrupt();+    }++    public void run() {+      while (!shutdown) {+        synchronized (this) {+          try {+            long currentTime = System.currentTimeMillis();++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+              cleanProjectFilesAndRemoveProject();+              lastProjectFileCleanTime = currentTime;+            }++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+              purgeRemovedProjects();+              lastProjectCleanTime = currentTime;+            }++            logger.info("Cleaner Thread completed. Waiting!!!!!");+            wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+          } catch (ProjectManagerException e) {+            logger.error(this.getName() + "failed due to: " + e.getMessage());+          } catch (InterruptedException e) {+            logger.info("Interrupted. Probably to shut down.");+          }+        }+      }+    }++    private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+      User deleter = new User(DELETER);+      long currentTime = System.currentTimeMillis();++      Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+      logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++      List<Project> projects = projectsById.values()+              .stream()+              .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+                      currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+              .collect(Collectors.toList());++      logger.info("Project files to clean: " + projects.size());++      for(Project project : projects) {+        logger.debug("Cleaning project files for project: " + project.getId());+        projectLoader.cleanProjectFiles(project);+        logger.debug("Removing project: " + project.getId());+        removeProject(project, deleter);+      }+    }++    private void purgeRemovedProjects() throws ProjectManagerException {+      long currentTime = System.currentTimeMillis();++      List<Project> inactiveProjects = projectLoader.fetchAllInactiveProjects();

Lets be consistent in naming.

  1. removed
  2. deleted
  3. inactive
  4. deactivated Lets use one everywhere consistently
mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";++    private boolean shutdown = false;+    private long lastProjectFileCleanTime = -1;+    private long lastProjectCleanTime = -1;++    public CleanerThread() {+      this.setName("ProjectManager-Cleaner-Thread");+      logger.info("Starting cleaner thread: " + this.getName());+    }++    @SuppressWarnings("unused")+    public void shutdown() {+      shutdown = true;+      this.interrupt();+    }++    public void run() {+      while (!shutdown) {+        synchronized (this) {+          try {+            long currentTime = System.currentTimeMillis();++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+              cleanProjectFilesAndRemoveProject();+              lastProjectFileCleanTime = currentTime;+            }++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+              purgeRemovedProjects();+              lastProjectCleanTime = currentTime;+            }++            logger.info("Cleaner Thread completed. Waiting!!!!!");+            wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+          } catch (ProjectManagerException e) {+            logger.error(this.getName() + "failed due to: " + e.getMessage());+          } catch (InterruptedException e) {+            logger.info("Interrupted. Probably to shut down.");+          }+        }+      }+    }++    private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+      User deleter = new User(DELETER);+      long currentTime = System.currentTimeMillis();++      Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+      logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++      List<Project> projects = projectsById.values()+              .stream()+              .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+                      currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+              .collect(Collectors.toList());++      logger.info("Project files to clean: " + projects.size());++      for(Project project : projects) {+        logger.debug("Cleaning project files for project: " + project.getId());+        projectLoader.cleanProjectFiles(project);+        logger.debug("Removing project: " + project.getId());+        removeProject(project, deleter);+      }+    }++    private void purgeRemovedProjects() throws ProjectManagerException {+      long currentTime = System.currentTimeMillis();++      List<Project> inactiveProjects = projectLoader.fetchAllInactiveProjects();+      logger.info("Inactive projects count: " + inactiveProjects.size());++      Set<Integer> purgedProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.PURGE);+      logger.info("Purged projects count: " + purgedProjectIds.size());++      List<Project> projectsToPurge = inactiveProjects+              .stream()+              .filter(project -> !purgedProjectIds.contains(project.getId()) &&+                      currentTime - project.getLastModifiedTimestamp() > removedProjectRetentionMs)+              .collect(Collectors.toList());++      logger.info("Projects to purge count: " + projectsToPurge.size());++      User deleter = new User(DELETER);

Singleton

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     logger.info("Project version retention is set to "         + projectVersionRetention); +    this.projectFilesRetentionMs =+            (props.getLong("project.files.retention", DEFAULT_RETENTION_MS));+    logger.info("Project files retention is set to "+            + projectFilesRetentionMs);++    this.removedProjectRetentionMs =+            (props.getLong("removed.project.retention", DEFAULT_RETENTION_MS));

Same as above; Also do you want to call it removed or deactivated as per your doc https://docs.google.com/document/d/1zOVZ_TvGEbfKhoIWpuELMmHr2r4Iw1IIQUx957pcM40/edit ?

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";++    private boolean shutdown = false;+    private long lastProjectFileCleanTime = -1;+    private long lastProjectCleanTime = -1;++    public CleanerThread() {+      this.setName("ProjectManager-Cleaner-Thread");+      logger.info("Starting cleaner thread: " + this.getName());+    }++    @SuppressWarnings("unused")+    public void shutdown() {+      shutdown = true;+      this.interrupt();+    }++    public void run() {+      while (!shutdown) {+        synchronized (this) {+          try {+            long currentTime = System.currentTimeMillis();++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+              cleanProjectFilesAndRemoveProject();+              lastProjectFileCleanTime = currentTime;+            }++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+              purgeRemovedProjects();+              lastProjectCleanTime = currentTime;+            }++            logger.info("Cleaner Thread completed. Waiting!!!!!");+            wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+          } catch (ProjectManagerException e) {+            logger.error(this.getName() + "failed due to: " + e.getMessage());+          } catch (InterruptedException e) {+            logger.info("Interrupted. Probably to shut down.");+          }+        }+      }+    }++    private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+      User deleter = new User(DELETER);+      long currentTime = System.currentTimeMillis();++      Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+      logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++      List<Project> projects = projectsById.values()

Q: projectsById is an update in memory map/cache on top of underlying mysql data is it ?

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";++    private boolean shutdown = false;+    private long lastProjectFileCleanTime = -1;+    private long lastProjectCleanTime = -1;++    public CleanerThread() {+      this.setName("ProjectManager-Cleaner-Thread");+      logger.info("Starting cleaner thread: " + this.getName());+    }++    @SuppressWarnings("unused")+    public void shutdown() {+      shutdown = true;+      this.interrupt();+    }++    public void run() {+      while (!shutdown) {+        synchronized (this) {+          try {+            long currentTime = System.currentTimeMillis();++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+              cleanProjectFilesAndRemoveProject();+              lastProjectFileCleanTime = currentTime;+            }++            if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+              purgeRemovedProjects();+              lastProjectCleanTime = currentTime;+            }++            logger.info("Cleaner Thread completed. Waiting!!!!!");+            wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+          } catch (ProjectManagerException e) {+            logger.error(this.getName() + "failed due to: " + e.getMessage());+          } catch (InterruptedException e) {+            logger.info("Interrupted. Probably to shut down.");+          }

Any other exception - this thread dies and we are forced to restart the svc ?

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     new XmlValidatorManager(prop);     loadAllProjects();     loadProjectWhiteList();++    CleanerThread cleanerThread = new CleanerThread();+    cleanerThread.start();+  }++  /**+   * Cleaner thread to clean projects. Runs every day.+  **/+  private class CleanerThread extends Thread {++    // runs every day+    private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+    private static final String DELETER = "azkaban";++    private boolean shutdown = false;

preferably volatile

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     logger.info("Project version retention is set to "         + projectVersionRetention); +    this.projectFilesRetentionMs =+            (props.getLong("project.files.retention", DEFAULT_RETENTION_MS));+    logger.info("Project files retention is set to "+            + projectFilesRetentionMs);++    this.removedProjectRetentionMs =+            (props.getLong("removed.project.retention", DEFAULT_RETENTION_MS));

Can you add some doc around each of the property

mudit1289

comment created time in 3 days

Pull request review commentFlipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup

 public ProjectManager(ProjectLoader loader, Props props) {     logger.info("Project version retention is set to "         + projectVersionRetention); +    this.projectFilesRetentionMs =+            (props.getLong("project.files.retention", DEFAULT_RETENTION_MS));

can you rename to project.files.retention.ms to avoid confusion with non-time based retention props such as project.version.retention

mudit1289

comment created time in 3 days

PR opened Flipkart/azkaban2

FDG-5837 : Azkaban Projects Cleanup
+259 -9

0 comment

4 changed files

pr created time in 3 days

release layer5io/meshery

v0.5.23

released time in 11 days

release layer5io/meshery

v0.5.22

released time in 11 days

release layer5io/meshery

v0.5.22

released time in 11 days

release layer5io/meshery

v0.5.22

released time in 11 days

release layer5io/meshery

v0.5.21

released time in 11 days