chandresh-pancholi/developer-to-engineer 2
👷 A reading list to help transform you from a developer to an engineer
chandresh-pancholi/engineering-blogs 1
A curated list of engineering blogs
Container-native workflows for Kubernetes.
chandresh-pancholi/awesome-go 0
A curated list of awesome Go frameworks, libraries and software
chandresh-pancholi/awesome-interview-questions 0
:octocat: A curated awesome list of lists of interview questions. Feel free to contribute! :mortar_board:
chandresh-pancholi/awesome-java 0
A curated list of awesome Java frameworks, libraries and software.
chandresh-pancholi/awesome-mongodb 0
:leaves: A curated list of awesome MongoDB resources, libraries, tools and applications
chandresh-pancholi/awesome-prometheus 0
A curated list of awesome Prometheus resources, projects and tools.
chandresh-pancholi/awesome-react-native 0
Awesome React Native components, news, tools, and learning material!
chandresh-pancholi/Awesome-React-Native-Education 0
Use this to learn React Native
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public Props fetchProjectProperty(Project project, String propsName) } } + public Set<Integer> fetchProjectIdsByEventType(EventType eventType) throws ProjectManagerException {+ QueryRunner runner = createQueryRunner();
Don't we need connection here like you did in fetchAllInactiveProjects?
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
private void cleanOlderProjectVersion(Connection connection, int projectId, } } + private static class ProjectIdsByEventTypeResultHandler implements+ ResultSetHandler<Set<Integer>> {+ private static final String SELECT_PROJECTS_BY_EVENT_TYPE =+ "SELECT distinct(project_id) from project_events WHERE `event_type`=?";
This will be costly as it will scan significant data. Should we keep this in the main sync block ?
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";++ private boolean shutdown = false;+ private long lastProjectFileCleanTime = -1;+ private long lastProjectCleanTime = -1;++ public CleanerThread() {+ this.setName("ProjectManager-Cleaner-Thread");+ logger.info("Starting cleaner thread: " + this.getName());+ }++ @SuppressWarnings("unused")+ public void shutdown() {
Who is calling this ? Worst case we need to register a shutdownhook isn't it ?
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public JdbcProjectLoader(Props props) { return projects; } + private List<Project> fetchAllInactiveProjects(Connection connection)+ throws ProjectManagerException {+ QueryRunner runner = new QueryRunner();++ ProjectResultHandler handler = new ProjectResultHandler();+ List<Project> projects = null;+ try {+ projects =+ runner.query(connection,+ ProjectResultHandler.SELECT_ALL_INACTIVE_PROJECTS, handler);++ for (Project project : projects) {
Common code w.r.t to fetchAllActiveProjects, can we avoid it
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public JdbcProjectLoader(Props props) { return projects; } + @Override+ public List<Project> fetchAllInactiveProjects() throws ProjectManagerException {+ Connection connection = getConnection();++ List<Project> projects = null;+ try {+ projects = fetchAllInactiveProjects(connection);
simply return here w/p projects variable ?
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";++ private boolean shutdown = false;+ private long lastProjectFileCleanTime = -1;+ private long lastProjectCleanTime = -1;++ public CleanerThread() {+ this.setName("ProjectManager-Cleaner-Thread");+ logger.info("Starting cleaner thread: " + this.getName());+ }++ @SuppressWarnings("unused")+ public void shutdown() {+ shutdown = true;+ this.interrupt();+ }++ public void run() {+ while (!shutdown) {+ synchronized (this) {
Did we use sync only for the wait() below? Can't we rather use sleep(ms) - this won't need sync ? My issue or doubt here is, the functionality being done is a bit heavy to be done under a single lock. Either remove sync or keep only necessary critical section with sync
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public void cleanOlderProjectVersion(int projectId, int version) } + @Override+ public void cleanProjectFiles(Project project) throws ProjectManagerException {+ // TODO Auto-generated method stub
UTs missing ?
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";++ private boolean shutdown = false;+ private long lastProjectFileCleanTime = -1;+ private long lastProjectCleanTime = -1;++ public CleanerThread() {+ this.setName("ProjectManager-Cleaner-Thread");+ logger.info("Starting cleaner thread: " + this.getName());+ }++ @SuppressWarnings("unused")+ public void shutdown() {+ shutdown = true;+ this.interrupt();+ }++ public void run() {+ while (!shutdown) {+ synchronized (this) {+ try {+ long currentTime = System.currentTimeMillis();++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+ cleanProjectFilesAndRemoveProject();+ lastProjectFileCleanTime = currentTime;+ }++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+ purgeRemovedProjects();+ lastProjectCleanTime = currentTime;+ }++ logger.info("Cleaner Thread completed. Waiting!!!!!");+ wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+ } catch (ProjectManagerException e) {+ logger.error(this.getName() + "failed due to: " + e.getMessage());+ } catch (InterruptedException e) {+ logger.info("Interrupted. Probably to shut down.");+ }+ }+ }+ }++ private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+ User deleter = new User(DELETER);+ long currentTime = System.currentTimeMillis();++ Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+ logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++ List<Project> projects = projectsById.values()+ .stream()+ .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+ currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+ .collect(Collectors.toList());++ logger.info("Project files to clean: " + projects.size());++ for(Project project : projects) {+ logger.debug("Cleaning project files for project: " + project.getId());
log info perhaps as we are cleaning up ?
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";++ private boolean shutdown = false;+ private long lastProjectFileCleanTime = -1;+ private long lastProjectCleanTime = -1;++ public CleanerThread() {+ this.setName("ProjectManager-Cleaner-Thread");+ logger.info("Starting cleaner thread: " + this.getName());+ }++ @SuppressWarnings("unused")+ public void shutdown() {+ shutdown = true;+ this.interrupt();+ }++ public void run() {+ while (!shutdown) {+ synchronized (this) {+ try {+ long currentTime = System.currentTimeMillis();++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+ cleanProjectFilesAndRemoveProject();+ lastProjectFileCleanTime = currentTime;+ }++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+ purgeRemovedProjects();+ lastProjectCleanTime = currentTime;+ }++ logger.info("Cleaner Thread completed. Waiting!!!!!");+ wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+ } catch (ProjectManagerException e) {+ logger.error(this.getName() + "failed due to: " + e.getMessage());+ } catch (InterruptedException e) {+ logger.info("Interrupted. Probably to shut down.");+ }+ }+ }+ }++ private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+ User deleter = new User(DELETER);+ long currentTime = System.currentTimeMillis();++ Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+ logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++ List<Project> projects = projectsById.values()+ .stream()+ .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+ currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+ .collect(Collectors.toList());++ logger.info("Project files to clean: " + projects.size());
can we log only if size is > 0
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";
Instead can't we simply have a private static final User deleter = new User(DELETER); ?
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";++ private boolean shutdown = false;+ private long lastProjectFileCleanTime = -1;+ private long lastProjectCleanTime = -1;++ public CleanerThread() {+ this.setName("ProjectManager-Cleaner-Thread");+ logger.info("Starting cleaner thread: " + this.getName());+ }++ @SuppressWarnings("unused")+ public void shutdown() {+ shutdown = true;+ this.interrupt();+ }++ public void run() {+ while (!shutdown) {+ synchronized (this) {+ try {+ long currentTime = System.currentTimeMillis();++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+ cleanProjectFilesAndRemoveProject();+ lastProjectFileCleanTime = currentTime;+ }++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+ purgeRemovedProjects();+ lastProjectCleanTime = currentTime;+ }++ logger.info("Cleaner Thread completed. Waiting!!!!!");+ wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+ } catch (ProjectManagerException e) {+ logger.error(this.getName() + "failed due to: " + e.getMessage());+ } catch (InterruptedException e) {+ logger.info("Interrupted. Probably to shut down.");+ }+ }+ }+ }++ private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+ User deleter = new User(DELETER);+ long currentTime = System.currentTimeMillis();++ Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+ logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++ List<Project> projects = projectsById.values()+ .stream()+ .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+ currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+ .collect(Collectors.toList());++ logger.info("Project files to clean: " + projects.size());++ for(Project project : projects) {+ logger.debug("Cleaning project files for project: " + project.getId());+ projectLoader.cleanProjectFiles(project);+ logger.debug("Removing project: " + project.getId());+ removeProject(project, deleter);+ }+ }++ private void purgeRemovedProjects() throws ProjectManagerException {+ long currentTime = System.currentTimeMillis();++ List<Project> inactiveProjects = projectLoader.fetchAllInactiveProjects();+ logger.info("Inactive projects count: " + inactiveProjects.size());++ Set<Integer> purgedProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.PURGE);+ logger.info("Purged projects count: " + purgedProjectIds.size());++ List<Project> projectsToPurge = inactiveProjects+ .stream()+ .filter(project -> !purgedProjectIds.contains(project.getId()) &&+ currentTime - project.getLastModifiedTimestamp() > removedProjectRetentionMs)+ .collect(Collectors.toList());++ logger.info("Projects to purge count: " + projectsToPurge.size());++ User deleter = new User(DELETER);+ for(Project project: projectsToPurge) {+ logger.debug("Purging project: " + project.getId());+ purgeProject(project, deleter);+ }+ } } private void loadAllProjects() { List<Project> projects; try { projects = projectLoader.fetchAllActiveProjects();+ logger.info("Active projects count: " + projects.size());
Do we need this? We have enough logging above
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";++ private boolean shutdown = false;+ private long lastProjectFileCleanTime = -1;+ private long lastProjectCleanTime = -1;
lastProjectCleanTime -> lastDeactivatedProjectCleanTime ?
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";++ private boolean shutdown = false;+ private long lastProjectFileCleanTime = -1;+ private long lastProjectCleanTime = -1;++ public CleanerThread() {+ this.setName("ProjectManager-Cleaner-Thread");+ logger.info("Starting cleaner thread: " + this.getName());+ }++ @SuppressWarnings("unused")+ public void shutdown() {+ shutdown = true;+ this.interrupt();+ }++ public void run() {+ while (!shutdown) {+ synchronized (this) {+ try {+ long currentTime = System.currentTimeMillis();++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+ cleanProjectFilesAndRemoveProject();+ lastProjectFileCleanTime = currentTime;+ }++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+ purgeRemovedProjects();+ lastProjectCleanTime = currentTime;+ }++ logger.info("Cleaner Thread completed. Waiting!!!!!");+ wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+ } catch (ProjectManagerException e) {+ logger.error(this.getName() + "failed due to: " + e.getMessage());+ } catch (InterruptedException e) {+ logger.info("Interrupted. Probably to shut down.");+ }+ }+ }+ }++ private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+ User deleter = new User(DELETER);+ long currentTime = System.currentTimeMillis();++ Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+ logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++ List<Project> projects = projectsById.values()+ .stream()+ .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+ currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+ .collect(Collectors.toList());++ logger.info("Project files to clean: " + projects.size());++ for(Project project : projects) {+ logger.debug("Cleaning project files for project: " + project.getId());+ projectLoader.cleanProjectFiles(project);+ logger.debug("Removing project: " + project.getId());+ removeProject(project, deleter);+ }+ }++ private void purgeRemovedProjects() throws ProjectManagerException {+ long currentTime = System.currentTimeMillis();++ List<Project> inactiveProjects = projectLoader.fetchAllInactiveProjects();+ logger.info("Inactive projects count: " + inactiveProjects.size());++ Set<Integer> purgedProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.PURGE);+ logger.info("Purged projects count: " + purgedProjectIds.size());++ List<Project> projectsToPurge = inactiveProjects+ .stream()+ .filter(project -> !purgedProjectIds.contains(project.getId()) &&+ currentTime - project.getLastModifiedTimestamp() > removedProjectRetentionMs)+ .collect(Collectors.toList());++ logger.info("Projects to purge count: " + projectsToPurge.size());
can you club these 3 logs into 1 and that too is projectsToPurge.size() > 0 to avoid spamming logs
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;
Shouldn't this be configurable as well ?
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";++ private boolean shutdown = false;+ private long lastProjectFileCleanTime = -1;+ private long lastProjectCleanTime = -1;++ public CleanerThread() {+ this.setName("ProjectManager-Cleaner-Thread");+ logger.info("Starting cleaner thread: " + this.getName());+ }++ @SuppressWarnings("unused")+ public void shutdown() {+ shutdown = true;+ this.interrupt();+ }++ public void run() {+ while (!shutdown) {+ synchronized (this) {+ try {+ long currentTime = System.currentTimeMillis();++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+ cleanProjectFilesAndRemoveProject();+ lastProjectFileCleanTime = currentTime;+ }++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+ purgeRemovedProjects();+ lastProjectCleanTime = currentTime;+ }++ logger.info("Cleaner Thread completed. Waiting!!!!!");+ wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+ } catch (ProjectManagerException e) {+ logger.error(this.getName() + "failed due to: " + e.getMessage());+ } catch (InterruptedException e) {+ logger.info("Interrupted. Probably to shut down.");+ }+ }+ }+ }++ private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+ User deleter = new User(DELETER);+ long currentTime = System.currentTimeMillis();++ Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+ logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++ List<Project> projects = projectsById.values()+ .stream()+ .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+ currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+ .collect(Collectors.toList());++ logger.info("Project files to clean: " + projects.size());
info(Project files to clean for: " + size + " projects")
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";++ private boolean shutdown = false;+ private long lastProjectFileCleanTime = -1;+ private long lastProjectCleanTime = -1;++ public CleanerThread() {+ this.setName("ProjectManager-Cleaner-Thread");+ logger.info("Starting cleaner thread: " + this.getName());+ }++ @SuppressWarnings("unused")+ public void shutdown() {+ shutdown = true;+ this.interrupt();+ }++ public void run() {+ while (!shutdown) {+ synchronized (this) {+ try {+ long currentTime = System.currentTimeMillis();++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+ cleanProjectFilesAndRemoveProject();+ lastProjectFileCleanTime = currentTime;+ }++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+ purgeRemovedProjects();+ lastProjectCleanTime = currentTime;+ }++ logger.info("Cleaner Thread completed. Waiting!!!!!");+ wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+ } catch (ProjectManagerException e) {+ logger.error(this.getName() + "failed due to: " + e.getMessage());+ } catch (InterruptedException e) {+ logger.info("Interrupted. Probably to shut down.");+ }+ }+ }+ }++ private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+ User deleter = new User(DELETER);+ long currentTime = System.currentTimeMillis();++ Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+ logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++ List<Project> projects = projectsById.values()+ .stream()+ .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+ currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+ .collect(Collectors.toList());++ logger.info("Project files to clean: " + projects.size());++ for(Project project : projects) {+ logger.debug("Cleaning project files for project: " + project.getId());+ projectLoader.cleanProjectFiles(project);+ logger.debug("Removing project: " + project.getId());+ removeProject(project, deleter);+ }+ }++ private void purgeRemovedProjects() throws ProjectManagerException {+ long currentTime = System.currentTimeMillis();++ List<Project> inactiveProjects = projectLoader.fetchAllInactiveProjects();
Lets be consistent in naming.
- removed
- deleted
- inactive
- deactivated Lets use one everywhere consistently
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";++ private boolean shutdown = false;+ private long lastProjectFileCleanTime = -1;+ private long lastProjectCleanTime = -1;++ public CleanerThread() {+ this.setName("ProjectManager-Cleaner-Thread");+ logger.info("Starting cleaner thread: " + this.getName());+ }++ @SuppressWarnings("unused")+ public void shutdown() {+ shutdown = true;+ this.interrupt();+ }++ public void run() {+ while (!shutdown) {+ synchronized (this) {+ try {+ long currentTime = System.currentTimeMillis();++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+ cleanProjectFilesAndRemoveProject();+ lastProjectFileCleanTime = currentTime;+ }++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+ purgeRemovedProjects();+ lastProjectCleanTime = currentTime;+ }++ logger.info("Cleaner Thread completed. Waiting!!!!!");+ wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+ } catch (ProjectManagerException e) {+ logger.error(this.getName() + "failed due to: " + e.getMessage());+ } catch (InterruptedException e) {+ logger.info("Interrupted. Probably to shut down.");+ }+ }+ }+ }++ private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+ User deleter = new User(DELETER);+ long currentTime = System.currentTimeMillis();++ Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+ logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++ List<Project> projects = projectsById.values()+ .stream()+ .filter(project -> !scheduledProjectIds.contains(project.getId()) &&+ currentTime - project.getLastModifiedTimestamp() > projectFilesRetentionMs)+ .collect(Collectors.toList());++ logger.info("Project files to clean: " + projects.size());++ for(Project project : projects) {+ logger.debug("Cleaning project files for project: " + project.getId());+ projectLoader.cleanProjectFiles(project);+ logger.debug("Removing project: " + project.getId());+ removeProject(project, deleter);+ }+ }++ private void purgeRemovedProjects() throws ProjectManagerException {+ long currentTime = System.currentTimeMillis();++ List<Project> inactiveProjects = projectLoader.fetchAllInactiveProjects();+ logger.info("Inactive projects count: " + inactiveProjects.size());++ Set<Integer> purgedProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.PURGE);+ logger.info("Purged projects count: " + purgedProjectIds.size());++ List<Project> projectsToPurge = inactiveProjects+ .stream()+ .filter(project -> !purgedProjectIds.contains(project.getId()) &&+ currentTime - project.getLastModifiedTimestamp() > removedProjectRetentionMs)+ .collect(Collectors.toList());++ logger.info("Projects to purge count: " + projectsToPurge.size());++ User deleter = new User(DELETER);
Singleton
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { logger.info("Project version retention is set to " + projectVersionRetention); + this.projectFilesRetentionMs =+ (props.getLong("project.files.retention", DEFAULT_RETENTION_MS));+ logger.info("Project files retention is set to "+ + projectFilesRetentionMs);++ this.removedProjectRetentionMs =+ (props.getLong("removed.project.retention", DEFAULT_RETENTION_MS));
Same as above; Also do you want to call it removed or deactivated as per your doc https://docs.google.com/document/d/1zOVZ_TvGEbfKhoIWpuELMmHr2r4Iw1IIQUx957pcM40/edit ?
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";++ private boolean shutdown = false;+ private long lastProjectFileCleanTime = -1;+ private long lastProjectCleanTime = -1;++ public CleanerThread() {+ this.setName("ProjectManager-Cleaner-Thread");+ logger.info("Starting cleaner thread: " + this.getName());+ }++ @SuppressWarnings("unused")+ public void shutdown() {+ shutdown = true;+ this.interrupt();+ }++ public void run() {+ while (!shutdown) {+ synchronized (this) {+ try {+ long currentTime = System.currentTimeMillis();++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+ cleanProjectFilesAndRemoveProject();+ lastProjectFileCleanTime = currentTime;+ }++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+ purgeRemovedProjects();+ lastProjectCleanTime = currentTime;+ }++ logger.info("Cleaner Thread completed. Waiting!!!!!");+ wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+ } catch (ProjectManagerException e) {+ logger.error(this.getName() + "failed due to: " + e.getMessage());+ } catch (InterruptedException e) {+ logger.info("Interrupted. Probably to shut down.");+ }+ }+ }+ }++ private void cleanProjectFilesAndRemoveProject() throws ProjectManagerException {+ User deleter = new User(DELETER);+ long currentTime = System.currentTimeMillis();++ Set<Integer> scheduledProjectIds = projectLoader.fetchProjectIdsByEventType(EventType.SCHEDULE);+ logger.info("Scheduled Projects count: " + scheduledProjectIds.size());++ List<Project> projects = projectsById.values()
Q: projectsById is an update in memory map/cache on top of underlying mysql data is it ?
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";++ private boolean shutdown = false;+ private long lastProjectFileCleanTime = -1;+ private long lastProjectCleanTime = -1;++ public CleanerThread() {+ this.setName("ProjectManager-Cleaner-Thread");+ logger.info("Starting cleaner thread: " + this.getName());+ }++ @SuppressWarnings("unused")+ public void shutdown() {+ shutdown = true;+ this.interrupt();+ }++ public void run() {+ while (!shutdown) {+ synchronized (this) {+ try {+ long currentTime = System.currentTimeMillis();++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectFileCleanTime) {+ cleanProjectFilesAndRemoveProject();+ lastProjectFileCleanTime = currentTime;+ }++ if (currentTime - CLEANER_THREAD_WAIT_INTERVAL_MS > lastProjectCleanTime) {+ purgeRemovedProjects();+ lastProjectCleanTime = currentTime;+ }++ logger.info("Cleaner Thread completed. Waiting!!!!!");+ wait(CLEANER_THREAD_WAIT_INTERVAL_MS);+ } catch (ProjectManagerException e) {+ logger.error(this.getName() + "failed due to: " + e.getMessage());+ } catch (InterruptedException e) {+ logger.info("Interrupted. Probably to shut down.");+ }
Any other exception - this thread dies and we are forced to restart the svc ?
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { new XmlValidatorManager(prop); loadAllProjects(); loadProjectWhiteList();++ CleanerThread cleanerThread = new CleanerThread();+ cleanerThread.start();+ }++ /**+ * Cleaner thread to clean projects. Runs every day.+ **/+ private class CleanerThread extends Thread {++ // runs every day+ private static final long CLEANER_THREAD_WAIT_INTERVAL_MS = 24 * 60 * 60 * 1000;+ private static final String DELETER = "azkaban";++ private boolean shutdown = false;
preferably volatile
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { logger.info("Project version retention is set to " + projectVersionRetention); + this.projectFilesRetentionMs =+ (props.getLong("project.files.retention", DEFAULT_RETENTION_MS));+ logger.info("Project files retention is set to "+ + projectFilesRetentionMs);++ this.removedProjectRetentionMs =+ (props.getLong("removed.project.retention", DEFAULT_RETENTION_MS));
Can you add some doc around each of the property
comment created time in 3 days
Pull request review commentFlipkart/azkaban2
FDG-5837 : Azkaban Projects Cleanup
public ProjectManager(ProjectLoader loader, Props props) { logger.info("Project version retention is set to " + projectVersionRetention); + this.projectFilesRetentionMs =+ (props.getLong("project.files.retention", DEFAULT_RETENTION_MS));
can you rename to project.files.retention.ms to avoid confusion with non-time based retention props such as project.version.retention
comment created time in 3 days