profile
viewpoint

Ask questionsTests' build output for netcoreapp3+ mixes the system under test with VsTest implementation details.

Description

.NET Core 3 changed the default behavior of a project's build output folder (see "Build copies dependencies"). For test projects, this has the unfortunate side effect of polluting the test projects' build output folder with test-running infrastructure. In addition to being visually jarring (which is not important), this changes the system under test. The system under test is no longer the one the user intended:

TestBuildOutput

Context in this image:

  1. CustomConvention.Tests is a test project.
  2. Fixie.TestAdapter is the adapter for the Fixie test framework.
  3. Fixie.Integration is the intended system under test.
  4. Mono.* is a dependency of the TestAdapter.
  5. Shouldly is merely an assertion library referenced by the test project.
  6. Everything else is the result of referencing the VsTest testhost infrastructure.

For instance, the popular Newtonsoft.Json.dll used by the VsTest infrastructure gets copied out here beside the system under test, which could negatively impact behavior if the system under test has its own reference to some other version of the assembly. This is just one example. Even if the user's desired version of Newtonsoft.Json happens to be the one present, the underlying problem remains: we're not testing what we think we are testing. The build output folder is a mix of the system under test, the test adapter's own dependencies, and VsTest testhost infrastructure.

There's hope that this mixing of multiple systems isn't truly necessary:

  1. We see that under a netcoreapp2.1 build, VsTest was able to perform discovery and execution without needing to mix multiple systems into one folder. I presume it was able to use the "deps" files to locate the particular test adapter/etc under the nuget cache, which makes sense, and run it from there.
  2. When debugging this custom TestAdapter, even under the .NET Core 3 target, the VS "Modules" screen shows that the actual testhost.dll loaded at runtime is the one under the nuget cache. So, it seems the VsTest files in our build output folder isn't strictly necessary for the actual runtime behavior, but that it's just an unfortunate side effect of .NET Core 3's change in build output behavior.

So, there are two questions:

  1. Is there anything consumers can do right now that would prevent the pollution of the build output folder while still giving VsTest the chance to locate the right testhost under the nuget cache folder? Any way to provoke a test project to output "deps" similar to a .NET Core 2 build? Do we somehow have control with the "additional probing paths" concept?
  2. Can VsTest or the test SDK nuget package itself be altered to avoid polluting the build output folder by default? I understand that recent optimizations forced *TestAdapter.dll to be present in the build output folder so that it can be detected, but could the VsTest-specific assets be suppressed at least?

It appears that the Test SDK's use of NuGet package dependencies essentially overwrites the system under test, and might be the root cause.

Steps to reproduce

Create a solution with a single test project which targets both netcoreapp2.1 and netcoreapp3.1. Build. Inspect the build output folders for both target frameworks.

Expected behavior

Build output folder contains the system under test.

Actual behavior

Build output folder contains a mix of the system under test, the test adapter's own dependencies, and the VsTest infrastructure.

microsoft/vstest

Answer questions plioi

Some of the implementation details in VsTest, as it prepares to ultimately invoke the test host, makes it look like the primary intended path is for the bin/ folder to NOT contain all these extra assemblies and support files, and that the mode everyone is experiencing was supposed to just be a fallback case.

useful!

Related questions

VSTest.console.exe fails due to connection timeout. hot 3
VSTest Task Fails even though all tests pass hot 2
Enabling code coverage leads to errors in child processes hot 1
Xunit failed tests are not discovered when rerun. hot 1
Could not load file or assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader - 16.0.2-preview hot 1
Support wildcards in filenames for vstest.console.exe hot 1
VsTest hangs (v.15.6.0) or fails immediately (v.16.1.0) in Azure DevOps Release Pipeline hot 1
Enabling code coverage leads to errors in child processes hot 1
Could not load file or assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader - 16.0.2-preview hot 1
Error "Could not load file or assembly" after changing Microsoft.Dynamics365.UIAutomation.Sample.dll with Powershell - vstest hot 1
[UWP] TestPlatform error running on VM in Azure. - vstest hot 1
Could not load file or assembly 'Microsoft.VisualStudio.ArchitectureTools.PEReader - 16.0.2-preview hot 1
SatelliteResourceLanguages property doesn't filter all resources when publishing test projects hot 1
`dotnet test` fails to find .NET Core 3.0 SDK when installed from script to custom location hot 1
VSTest.console.exe fails due to connection timeout. hot 1
Github User Rank List