FuseSoC currently does one (and after #391 two) dependency resolution runs, and expects to be able to fully resolve all dependencies in that run.

With generators adding dynamicity to the picture, we should be looking into an iterative dependency resolution as long as new information becomes available.

What I'm looking for is an algorithm like this:

core_db = collect_core_files()
dep_tree = solve_dependency_tree(core_db, entrypoint)

while (True):
	# Perform all operations which might produce new cores
	changed = False
	generators = get_generator_list(dep_tree)
	for g in generators: 
		new_cores = run_generator_and_get_new_cores(g)
		if new_cores:
			core_db += new_cores
			changed = True	

	# In the future: Expand to look at certain remote locations for new cores
	
	# Resolve dependency tree again to take new information into account.
	dep_tree_before = dep_tree
	dep_tree = solve_dependency_tree(core_db, entrypoint)
	if dep_tree_before == dep_tree:
		break

if not is_completely_resolved(dep_tree):
	exit("Unable to resolve all dependencies.")

This algorithm isn't guaranteed to stabilize, so we need to add additional exit conditions, e.g. abort after n tries. Equally, there is room for some optimization (e.g. avoid the second resolution if generators didn't produce new cores).

Use cases for this new approach:

• Cores can depend on cores which are produced by a generator, but don't exist anywhere else. This is actually something we're likely to need in OpenTitan, and effectively solves #235 through generators (more on that once I've finished thinking about it).
• We can expand the search scope step-by-step. For example: First, only search local cores, then also search cores in a certain remote location (librecores!). This will keep dependency resolution fast for local use cases.

Using this code, you can test the difference between using delay() and using Helios. If you run the regular example, the button will not start as soon as you push it.

XDG isn't applicable to Windows at all, it's a Linux-only thing.

I'm not a big fan of moving sruff without a good reason since we need to add code for the migration parh and probably lose the backwards compatibility.

With that said, what does the xdg spec say about this on windows? ~/.config is the fallback if it can't find the xdg definitions. I' ve been considering to use the xdg_utils python package to save a few lines of code and add consistency but it has been low priority

Hello, At least on my windows installation the initial fusesoc.conf file is located in the following Path: %HOMEPATH%\.config\fusesoc
Regards, Markus

Seems like many other applications originating from the Unix world have the same problem :-) I filed https://github.com/olofk/fusesoc/issues/448 to get this sorted properly, but for now, let's at least document what we have. Thank you!

Currently, FuseSoC stores its configuration file under Windows at ~/.config, which resolves normally to something like C:\Users\john\.config.

Until #447 we documented we'd put the config into LOCALAPPDATA, but actually put it into the home directory.

https://github.com/olofk/fusesoc/blob/24350b2d10a8b27819d84210554654baff6cd877/fusesoc/main.py#L203-L204

Using the home directory is very Unix-y, the more typical variant for Windows would be %APPDATA% (roaming, i.e. per user) or %LOCALAPPDATA% (per user and machine).

We should

• Decide on the use of LOCALAPPDATA vs APPDATA. (Where does pip place its packages? Do they roam?)
• Change the config file location.
• Include code to move the config file over if it is found in the old path.

You're welcome @imphil and thanks for your explanation!

From a user perspective I agree, that %LOCALAPPDATA% is a better location as this would keep my %HOMEPATH% tidy. My %HOMEPATH% directory is cluttered with a lot folders from many different tools (for example .eclipse, .atom, .mozilla, ...). It seems, that using %HOMEPATH% is a common practice.

Thanks for this documentation update @hackathlete!

Your fix is correct, we use os.path.expanduser() on all platforms, which is documented for Windows at https://docs.python.org/3/library/os.path.html#os.path.expanduser.

I'm wondering, however, if the current behavior is actually what we want, or if we should put the config file into LOCALAPPDATA, where it probably belongs?

(In that case, we should still take this PR for now, but do a follow-up to migrate the config file to LOCALAPPDATA.)

Hello, At least on my windows installation the initial fusesoc.conf file is located in the following Path: %HOMEPATH%.config\fusesoc
Regards, Markus

This test is only to document the current behavior, which will change in #391. I want to ensure, however, that we have a very clear picture on what changes and ensure that we don't miss anything, so I'd like to get this test change in first.

The position attriute is meant to insert files generated by the core at a specific location: after the chosen core, at the beginning of the file list, or at the end. This behavior was not tested so far. Add a test for it.

Seems like I dropped @benreynwar's actual fix and only picked the test changes in my rebasing action. It is a mystery to me why CI didn't pick that up, but the most likely explanation is that the test ran actually not against the PR, but against master.

I added the changes back in https://github.com/olofk/fusesoc/pull/444. Sorry for the confusion (well, I mostly confused myself).

Right now, we run pull request CI jobs in the context of the olofk/fusesoc repository. We did this to be able to create Check runs, which show the pytest results in the GitHub UI. There are multiple problems with that:

First, the runs actually didn't contain any data. That's probably fixable if we found the right configuration regarding test result files and artifacts.

Second, and more importantly, it seems that the "old" test code was run, i.e. pytest didn't pick up test changes done in the PR. That's hard to believe, but also not very useful, as PRs often modify code and tests at the same time. There might be some misconfiguration going on somewhere, but I didn't debug it fully.

Finally, the test result reporting action we used is actually rather slow (significantly slower than the test runs themselves), as it creates and then runs a Docker container.

In this commit, we try a different approach and annotate the source code with test results.

Oh man. CI in this PR (and local testing) passed, but after rebasing this PR into master CI fails (https://github.com/olofk/fusesoc/runs/1417908199). Let me figure out what is going on ...

We currently get a failing CI test for each PR:

This is due to AppVeyor not finding a config file any more (which is expected).

@olofk can you deactive the AppVeyor integration in the GH settings of this repository?

Nice! we now actually get test results inline in the code, e.g.

Currently, since we do matrix testing, each individual platform shows up and identical failures are not merged. That's not super beautiful, but probably better to show too much than too little. And not worth going in and developing a merging solution on our own, I'm sure someone will come up with something like that in the not too distant future.

Right now, we run pull request CI jobs in the context of the olofk/fusesoc repository. We did this to be able to create Check runs, which show the pytest results in the GitHub UI. There are multiple problems with that:

First, the runs actually didn't contain any data. That's probably fixable if we found the right configuration regarding test result files and artifacts.

Second, and more importantly, it seems that the "old" test code was run, i.e. pytest didn't pick up test changes done in the PR. That's hard to believe, but also not very useful, as PRs often modify code and tests at the same time. There might be some misconfiguration going on somewhere, but I didn't debug it fully.

Finally, the test result reporting action we used is actually rather slow (significantly slower than the test runs themselves), as it creates and then runs a Docker container.

In this commit, we try a different approach and annotate the source code with test results.

Here we go. I updated this PR on top of @benreynwar's test improvements, and did a bit more digging around the position attribute of generators. There are three things which need addressing:

• position is untested so far.
• position will most likely be unsupported/not needed any more with this change. I need to properly test and document that.
• Somehow CI is failing, but not for me locally. Could be a Python version issue, I'll do some digging.

Recent refactoring of the Edalizer class introduced a bug where the files in the generated .eda are not in a dependency-consistent order.

This fixes this bug, by keeping track of how the generated cores are positioned in the ordered list of cores.

OK, here we go. I updated this PR with a version that contains addresses my review comments and a couple tiny fixes, but is otherwise Ben's code. Let's get this in to document the current state of affairs regarding generators.

I do have code ready for #391 to update this test with the slightly different behavior we get in this PR. This should make it much easier to see what is actually changing.