profile
viewpoint
John Starks jstarks @Microsoft Microsoft OS engineering lead for containers, virtualization, WSL

dotnet/Docker.DotNet 1201

:whale: .NET (C#) Client Library for Docker API

jstarks/npiperelay 175

npiperelay allows you to access Windows named pipes from WSL

Azure/go-ansiterm 55

Go package for ANSI terminal emulation in Windows

jstarks/containerd 0

An open and reliable container runtime

jstarks/cri 0

Containerd Plugin for Kubernetes Container Runtime Interface

jstarks/curl 0

A command line tool and library for transferring data with URL syntax, supporting HTTP, HTTPS, FTP, FTPS, GOPHER, TFTP, SCP, SFTP, SMB, TELNET, DICT, LDAP, LDAPS, FILE, IMAP, SMTP, POP3, RTSP and RTMP. libcurl offers a myriad of powerful features

jstarks/diskspd 0

DISKSPD is a storage load generator / performance test tool from the Windows/Windows Server and Cloud Server Infrastructure Engineering teams

jstarks/distribution 0

The Docker toolset to pack, ship, store, and deliver content

jstarks/docker 0

Docker - the open-source application container engine

jstarks/Docker-PowerShell 0

PowerShell Module for Docker

created tagjstarks/npiperelay

tagv0.1.0

npiperelay allows you to access Windows named pipes from WSL

created time in 3 days

push eventjstarks/npiperelay

Stuart Leeks

commit sha 57eaf05f97e4f7dfb7d16a12dfc30f93d1211116

Add go.mod

view details

Stuart Leeks

commit sha 069a62cc29da59586ea667f4a3ae04d9cdd6ea1e

go mod vendor

view details

Stuart Leeks

commit sha 877c880fb5fc8e0b4d97208c917550463b56a104

Add GitHub action to build and release

view details

John Starks

commit sha 4997d3bf5f2e512608b39b51f37412206758051b

Merge pull request #14 from stuartleeks/sl/release-on-tag Create a release when a tag is pushed

view details

push time in 3 days

pull request commentmalxau/yori

dir: Support -color even with minimal output (-m)

Oh cool, I didn't know about sdir, that's quite nice. I was using dir -m -color for a kind of ls equivalent, where I don't want all the metadata.

By the way, is there a .profile/.bashrc equivalent for yori to set up aliases and things?

jstarks

comment created time in 15 days

issue openedrust-analyzer/rust-analyzer

rust-analyzer is broken on Alpine Linux

Alpine uses musl. Rust-analyzer in VSCode started shipping a glibc linked binary recently.

NixOS had the same issue (#4542), but the fix only applied to NixOS, apparently.

This is pretty disappointing given that Rust 1.44 finally added support for proc macros in Alpine... but now rust-analyzer is broken.

created time in 16 days

PR opened malxau/yori

dir: Support -color even with minimal output (-m)
+60 -47

0 comment

1 changed file

pr created time in 16 days

create barnchjstarks/yori

branch : dir_color_minimal

created branch time in 16 days

issue commentmicrosoft/winrt-rs

From<HString> for String can panic

It's interesting that there appears to be no TryFrom<OsString> for String. Not sure if it's an oversight or if there's some principled reason. It might be good to understand why that is before adding TryFrom<HString>.

jstarks

comment created time in 19 days

issue commentmicrosoft/winrt-rs

From<HString> for String can panic

Certainly it would be reasonable to have a method to_string_lossy on HString, though (like OsStr has).

jstarks

comment created time in 19 days

issue commentmicrosoft/winrt-rs

From<HString> for String can panic

That will make it impossible to get the correct file names when enumerating a folder, for example.

jstarks

comment created time in 19 days

issue openedmicrosoft/winrt-rs

From<HString> for String can panic

String in Rust requires that the underlying buffer be a valid UTF-8 string. HSTRING in Windows does not require that the underlying buffer be a valid UTF-16 string--it can be any sequence of 16-bit integers, including unmatched surrogates. Therefore, it's not always possible to translate an HSTRING into a String (but it is, of course, always valid to go the other direction).

However, we have:

impl<'a> From<&'a HString> for String {
    fn from(hstring: &HString) -> Self {
        String::from_utf16(hstring.as_wide()).unwrap()
    }
}

This will panic if the HSTRING contains non-UTF-16 characters.

This can happen, for example, if a user has a file with a non-UTF-16 name and calls one of the WinRT directory enumeration APIs.

It seems that the fix for this would be to implement From<HString> for OsString instead, as OsString can encode any Windows wide string, not just valid Unicode strings. The user can then try to convert it to a String or &str dynamically, falling back to some reasonable behavior (or panicking!) as appropriate.

This will make using HString more annoying for callers, but there's no real alternative I can see.

created time in 19 days

issue commentgolang/go

all: port to Windows/arm64

This sounds promising. I'll check on what options are available and if there are licensing implications.

vielmetti

comment created time in a month

issue commentgolang/go

all: port to Windows/arm64

We (the OS team at Microsoft) would love to see this, too, and we would be open to contributing the code for the port. Unfortunately, we're not in a position to commit to long-term maintenance of the code or associated builder right now, so we are hesitant to send out any patches.

If Google or the Golang community would be interesting in taking on the ongoing maintenance cost, we would be happy to collaborate on the development.

vielmetti

comment created time in a month

pull request commentmalxau/yori

Add additional keyboard navigation from Bash

Thanks!

jstarks

comment created time in a month

PR opened malxau/yori

Fix build on newer versions of VC++
+3 -2

0 comment

1 changed file

pr created time in a month

create barnchjstarks/yori

branch : build

created branch time in a month

PR opened malxau/yori

Add additional keyboard navigation from Bash

This set of commits adds additional input keyboard navigation commands:

  • Ctrl-B, Ctrl-F: aliases for Left, Right
  • Ctrl-D: alias for DEL (when the input is non-empty, otherwise it's still EXIT)
  • Ctrl-K: removes to the end of input
  • Ctrl-Y: pastes string that was removed by most recent Ctrl-K
+80 -16

0 comment

3 changed files

pr created time in a month

create barnchjstarks/yori

branch : bash_keyboard

created branch time in a month

pull request commentmicrosoft/go-winio

Update to use upstream sddl/SecurityAttribute but retain old exported functions

What does this change actually solve?

katiewasnothere

comment created time in a month

Pull request review commentmicrosoft/go-winio

Update to use upstream sddl/SecurityAttribute but retain old exported functions

 func makeServerPipeHandle(path string, sd []byte, c *PipeConfig, first bool) (sy 	if err := rtlDosPathNameToNtPathName(&path16[0], &ntPath, 0, 0).Err(); err != nil { 		return 0, &os.PathError{Op: "open", Path: path, Err: err} 	}-	defer localFree(ntPath.Buffer)+	defer windows.LocalFree(windows.Handle(ntPath.Buffer)) 	oa.ObjectName = &ntPath  	// The security descriptor is only needed for the first pipe. 	if first { 		if sd != nil {-			len := uint32(len(sd))+			len := sd.Length() 			sdb := localAlloc(0, len)-			defer localFree(sdb)-			copy((*[0xffff]byte)(unsafe.Pointer(sdb))[:], sd)+			defer windows.LocalFree(windows.Handle(sdb))+			copy((*[0xffff]byte)(unsafe.Pointer(sdb))[:len], (*[0xffff]byte)(unsafe.Pointer(sd))[:len])

We're fairly inconsistent about following the Go requirements around the use of Go pointers in FFI calls--would we face any possible object lifetime issues by doing this?

katiewasnothere

comment created time in a month

issue commentgolang/go

runtime: exit status 0xC0000005 from test process on windows-amd64-longtest builder

Interestingly, according to the Mm team, mappings and ReadFile should be coherent as long as the file was opened cached, i.e. without FILE_NO_INTERMEDIATE_BUFFERING. The fact that they appear not to be in this case may be an OS bug.

We (well, not me, but others here in Windows) will try to get a local repro under stress. In the meantime, it sounds like flushing the mapping explicitly is a reasonable workaround.

Thanks for looping me in. @aclements, on which OS build are you reproducing this?

bcmills

comment created time in a month

issue commentgolang/go

runtime: exit status 0xC0000005 from test process on windows-amd64-longtest builder

It looks like MiCreateImageFileMap flushes the data section (i.e. waits for modified pages to be written to the backing file) before doing any reads. So if the MiReadImageHeaders read is the first one in the CreateProcess path, then I wouldn't expect any coherence issues in that path.

Let me see if I can get someone from the Mm team to chime in.

bcmills

comment created time in a month

issue commentgolang/go

runtime: exit status 0xC0000005 from test process on windows-amd64-longtest builder

It seems like the most likely theory at this point is that the Windows loader uses read/write IO for at least some things (even if it memory maps the segments themselves), and that omitting the FlushViewOfFile causes that to be incoherent, which causes the loader to crash.

That would be surprising to me. I would expect that CreateProcess would construct an image section for the file, which I would expect to implicitly flush any remaining data section (and fail if there were any existing writable mappings). I'm not an expert in this area, and I did not read the code, so this is just my intuition.

However, this is not a problem. From @zx2c4's analysis, it sounded like the corruption was introduced earlier in the chain. If the Go linker is calling CopyFile (which calls NtReadFile) without calling FlushViewOfFile, then the copied binary could be corrupted due to incoherence well before CreateProcess is called on it.

bcmills

comment created time in a month

issue commentgolang/go

runtime: exit status 0xC0000005 from test process on windows-amd64-longtest builder

I'm not certain about the coherence question (based on the kernel data structures at play, I believe map+unmap+map (even with intervening close+open), will be coherent, but I can't speak from a position of authority).

But to @aclements's point, there are several code paths in the Windows loader that handle an access violation (or other exception) and call NtTerminateProcess without raising again. So it's certainly possible that a corrupt binary will exit without producing a crash dump. Unfortunately.

bcmills

comment created time in a month

push eventjstarks/hcsshim

John Starks

commit sha b2d3d15f62d3cc241cc27af4e98f19829e5f0334

cim: Add sparse stream support

view details

John Starks

commit sha 88285f452a943a62f0683384328605445bc0f68b

cim: Always provide a security descriptor

view details

John Starks

commit sha 1ddf6f4325ad227445104e32502c01a978dbe07b

cim wip

view details

John Starks

commit sha 02e42ffae5856945de57eb5a5faaf3128285addc

missing file

view details

push time in a month

issue commentgolang/go

os: Symlink on Windows with a to-be-created directory path silently creates a link of the wrong type

I don't have any creative solutions for this problem, unfortunately. We run into a similar issue when trying to determine what symlink type to create when creating symbolic links in WSL. If the target doesn't exist, or if the target type changes after creating the symlink, then we get it wrong. This breaks symlink accesses from Windows (but not from WSL since we interpret the symlinks there, ignoring the type flag).

Ideally we would change Windows to not require this distinction, but so far we don't think that's practical.

bcmills

comment created time in a month

issue commentmalxau/yori

Ctrl-left mutates the command line

Ha, yes. I feel like bash and Yori (and PowerShell) do the right thing, or at least what I'm used to. I thing DOS/Windows got it wrong by leaving (most) argument parsing up to the app (related problem: Yori builtins appear to support ~ meaning %USERPROFILE%, but the expansion is not done by the shell itself so I can't actually rely on it).

I think notably because PowerShell gets away with it, probably Yori can too.

But regardless of the string that is passed to CreateProcess, a more clever parsing algorithm could preserve the spaces during tab completion and cursor navigation.

jstarks

comment created time in a month

issue commentmalxau/yori

Ctrl-left mutates the command line

Cool, this is better, thanks! Ctrl-left will still collapse spaces, though--should that be made conditional too?

jstarks

comment created time in a month

issue commentmalxau/yori

Ctrl-left mutates the command line

Thanks. Yes, expansion still seems undesirable to me, but at least with your fix it's easier to tell what's going on.

jstarks

comment created time in a month

issue openedmalxau/yori

Ctrl-left mutates the command line

The Ctrl-Left functionality to move the cursor between words also expands environment variables and trims spaces. However, it doesn't actually re-render the command line immediately, so the mutations only show up after a subsequent editing command.

This can lead to some confusing and undesirable behavior, as seen in this video, where I use Ctrl-Left to jump past an environment variable and then press Del to delete the first character (which should be %):

yori

created time in a month

issue openedmalxau/yori

Recursive make considered harmful

http://www.real-linux.org.uk/recursivemake.pdf

Builds are slow (or perhaps I am just impatient) because nmake runs 3 times for every directory, even if nothing has changed.

Would a change to centralize the build directives for all components into one place be reasonable? Perhaps via includes to still allow each directory to keep its build rules locally?

created time in a month

PR opened malxau/yori

pushd: With no arguments, swap curdir and top of stack

In bash and csh, running pushd with no arguments will swap the current directory and the top of the stack. This is quite useful for rapidly switching between two directories without opening a second terminal window.

This change adds this feature to Yori.

+49 -25

0 comment

1 changed file

pr created time in a month

create barnchjstarks/yori

branch : pushd_noargs

created branch time in a month

fork jstarks/yori

Yori is a CMD replacement shell that supports backquotes, job control, and improves tab completion, file matching, aliases, command history, and more.

http://www.malsmith.net/yori/

fork in a month

Pull request review commentmicrosoft/hcsshim

Add virtual NUMA support

 func SpecToUVMCreateOpts(ctx context.Context, s *specs.Spec, id, owner string) ( 		lopts.StorageQoSBandwidthMaximum = ParseAnnotationsStorageBps(ctx, s, annotationStorageQoSBandwidthMaximum, lopts.StorageQoSBandwidthMaximum) 		lopts.StorageQoSIopsMaximum = ParseAnnotationsStorageIops(ctx, s, annotationStorageQoSIopsMaximum, lopts.StorageQoSIopsMaximum) 		lopts.PreferredRootFSType = parseAnnotationsPreferredRootFSType(ctx, s.Annotations, annotationPreferredRootFSType, lopts.PreferredRootFSType)+		lopts.DefaultNUMA = parseAnnotationsBool(ctx, s.Annotations, annotationDefaultNUMA, lopts.DefaultNUMA)+		if v, ok := s.Annotations[annotationVirtualNodeSettings]; ok {+			lopts.NUMATopologyJSON = v

I thought we had discussed this internally and decided to go with JSON. But then, I thought the suggestion was an annotation specifying a separate file, which reportedly is how Kata handles this kind of thing.

dcantah

comment created time in 2 months

Pull request review commentmicrosoft/hcsshim

Add GMSA support for V2 process isolated containers

+// +build windows++package hcsoci++import (+	"context"+	"encoding/json"+	"fmt"++	"github.com/Microsoft/hcsshim/internal/hcs"+	"github.com/Microsoft/hcsshim/internal/log"+	hcsschema "github.com/Microsoft/hcsshim/internal/schema2"+)++// CCGState stores a containers Container Credential Guard state. Used when+// closing a container to be able to release the instance.+type CCGState struct {+	// ID of container that instance belongs to.+	id string+}++// Release calls into hcs to remove the ccg instance. These do not get cleaned up automatically+// they MUST be explicitly removed with a call to ModifyServiceSettings. The instances will persist+// unless vmcompute.exe exits or they are removed manually as done here.+func (instance *CCGState) Release(ctx context.Context) error {+	if err := removeCredentialGuard(ctx, instance.id); err != nil {+		log.G(ctx).WithError(err).WithField("containerID", instance.id).Warn("failed to remove Container Credential Guard instance")+		return err+	}+	return nil+}++// CreateCredentialGuard creates a container credential guard and returns the state object to be placed in a v2 container doc.+func CreateCredentialGuard(ctx context.Context, ID, credSpec string, uvm bool) (*hcsschema.ContainerCredentialGuardState, error) {+	log.G(ctx).WithField("containerID", ID).Debug("creating container credential guard")+	// V2 schema ccg setup a little different as its expected to be passed+	// through all the way to the gcs. Can no longer be enabled just through+	// a single property. The flow is as follows+	// ------------------------------------------------------------------------+	// 1. Call HcsModifyServiceSettings with a ModificationRequest set with a+	// ContainerCredentialGuardAddInstanceRequest. This is where the cred spec+	// gets passed in. Transport either "LRPC" (Argon) or "HvSocket" (Xenon).+	// 2. Query the instance with a call to HcsGetServiceProperties with the+	// PropertyType "ContainerCredentialGuard". This will return all instances+	// 3. Parse for the id of our container to find which one correlates to the+	// container we're building the doc for, then add to the v2 doc.+	// 4. If xenon container the hvsocketconfig will need to be attached BEFORE+	// being created. It must be in the doc itself as we do not support hot adding+	// service table entries. This is currently a blocker for adding support for+	// hyper-v gmsa.+	transport := "LRPC"+	if uvm {+		transport = "HvSocket"+	}+	req := hcsschema.ModificationRequest{+		PropertyType: "ContainerCredentialGuard",+		Settings: &hcsschema.ContainerCredentialGuardOperationRequest{+			Operation: hcsschema.AddInstance,+			OperationDetails: &hcsschema.ContainerCredentialGuardAddInstanceRequest{+				Id:             ID,+				CredentialSpec: credSpec,+				Transport:      transport,+			},+		},+	}+	if err := hcs.ModifyServiceSettings(ctx, req); err != nil {+		log.G(ctx).WithError(err).WithField("containerID", ID).Warn("failed to generate container credential guard instance")+		return nil, err+	}++	q := hcsschema.PropertyQuery{+		PropertyTypes: []hcsschema.PropertyType{hcsschema.PTContainerCredentialGuard},+	}+	serviceProps, err := hcs.GetServiceProperties(ctx, q)+	if err != nil {+		log.G(ctx).WithError(err).WithField("containerID", ID).Warn("failed to retrieve container credential guard instances")+		return nil, err+	}++	ccgState, err := unmarshalCCGInstances(serviceProps, ID)+	if err != nil {+		return nil, err+	}+	return ccgState, nil+}++// Checks to see if doc has non-nil ccg field+func hasCCG(doc *hcsschema.Container) bool {+	if doc.ContainerCredentialGuard != nil {+		return true+	}+	return false+}++// Removes a ContainerCredentialGuard instance by container ID.+func removeCredentialGuard(ctx context.Context, ID string) error {+	log.G(ctx).WithField("containerID", ID).Debug("removing container credential guard")++	req := hcsschema.ModificationRequest{+		PropertyType: "ContainerCredentialGuard",+		Settings: &hcsschema.ContainerCredentialGuardOperationRequest{+			Operation: hcsschema.RemoveInstance,+			OperationDetails: &hcsschema.ContainerCredentialGuardRemoveInstanceRequest{+				Id: ID,+			},+		},+	}+	if err := hcs.ModifyServiceSettings(ctx, req); err != nil {+		return err+	}+	return nil+}++// Unmarshals a ServiceProperties struct if the only property queried was 'ContainerCredentialGuard'+// Returns the ContainerCredentialGuardInstance matching the id passed in.+func unmarshalCCGInstances(sp *hcsschema.ServiceProperties, ID string) (*hcsschema.ContainerCredentialGuardState, error) {+	if len(sp.Properties) != 1 {+		return nil, fmt.Errorf("wrong number of service properties present")+	}+	// Properties is []interface{}

Can we make Properties be []RawJson instead?

dcantah

comment created time in 3 months

Pull request review commentmicrosoft/hcsshim

Fix LCOW SCSI layer mounting

 func MountContainerLayers(ctx context.Context, layerFolders []string, guestRoot 			// fall back to SCSI. 			uvmPath, err = uvm.AddVPMEM(ctx, layerPath) 			if err == uvmpkg.ErrNoAvailableLocation || err == uvmpkg.ErrMaxVPMEMLayerSize {-				log.G(ctx).WithError(err).Debug("falling back to SCSI for lcow layer addition")+				log.G(ctx).WithError(err).Debug("falling back to SCSI for LCOW layer addition") 				sm, err := uvm.AddSCSILayer(ctx, layerPath) 				if err != nil {-					return "", err+					return "", fmt.Errorf("failed to add SCSI layer: %s", err) 				} 				uvmPath = sm.UVMPath+			} else if err != nil {+				return "", fmt.Errorf("failed to add VPMEM layer: %s", err) 			}-			if err == nil {-				layersAdded = append(layersAdded, layerPath)-				lcowUvmLayerPaths = append(lcowUvmLayerPaths, uvmPath)-			}-		}-		if err != nil {-			return "", err+			layersAdded = append(layersAdded, layerPath)+			lcowUvmLayerPaths = append(lcowUvmLayerPaths, uvmPath) 		} 	}  	hostPath := filepath.Join(layerFolders[len(layerFolders)-1], "sandbox.vhdx") 	containerScratchPathInUVM := ospath.Join(uvm.OS(), guestRoot)+	log.G(ctx).WithField("hostPath", hostPath).Debug("mounting scratch VHD") 	scsiMount, err := uvm.AddSCSI(ctx, hostPath, containerScratchPathInUVM, false) 	if err != nil {-		return "", err+		return "", fmt.Errorf("failed to add SCSI scratch VHD")

We lost err

kevpar

comment created time in 3 months

more