cython-hidapi is a different project. https://github.com/trezor/cython-hidapi

Just wondering which API are you referring to. Are you talking about cython-hidapi or hidapi itself? I will say the C HIDAPI APIs talk about unsigned char *data.

https://github.com/libusb/hidapi/blob/master/hidapi/hidapi.h

int  HID_API_EXPORT HID_API_CALL hid_write(hid_device *dev, const unsigned char *data, size_t length);
int HID_API_EXPORT HID_API_CALL hid_read_timeout(hid_device *dev, unsigned char *data, size_t length, int milliseconds);
int  HID_API_EXPORT HID_API_CALL hid_read(hid_device *dev, unsigned char *data, size_t length);
int HID_API_EXPORT HID_API_CALL hid_send_feature_report(hid_device *dev, const unsigned char *data, size_t length);
int HID_API_EXPORT HID_API_CALL hid_get_feature_report(hid_device *dev, unsigned char *data, size_t length);
int HID_API_EXPORT HID_API_CALL hid_get_input_report(hid_device *dev, unsigned char *data, size_t length);

The recv call returns a list of ints, one int for each byte in the data received from the device.

Consider the case where the data is a structure of strings floats and ints, or pixel data from an imaging sensor that you intend to pass on to other hardware.

Your hardware sent binary data, and the python hidapi gives it to your as list ints, an int per byte, which you now have to convert back to a bytearray, at the cost of copying and repacking the data.

While the list of ints might be okay for some use cases, it is not faithful to the data and by doing that, ignores and even vicerates the more general set of use cases.

Since, the authors are unwilling to correct this api because (on the thesis that it is already deployed), the next best solution is provide another call that does return the data faithful to what was sent.

Please think about this seriously. There is no point to defending a broken api. Users will simply replace it.

Okay here is the history of libusb HID Windows backend. https://github.com/libusb/libusb/blob/master/libusb/os/windows_winusb.c#L3452

It was actually based on the libusb-win32 1.0 branch. https://github.com/mcuee/libusb-win32/blob/master/libusb1/src/dll/usbi_backend_hid.c#L287

From Pete Batard's email dated 6-April-2010.

Yes, that is expected, as the HID API of Windows does not provide the ability to actually read the report descriptor from the device, but will only provide the input/output/feature report sizes (in bytes) through the HIDP_CAPS struct (http://msdn.microsoft.com/en-us/library/ms899397.aspx). AFAIK, this is about as far as we can get with regards to getting HID descriptor report data on Windows.

Thus, the report descriptor provided by libusb on Windows is a "fake" report, reconstructed from the sizes, and will pretty much always be much shorted than the Linux version. At the end of this e-mail, I'm providing the code that builds this fake report for reference (which I actually picked from libusb-win32).

Diving into the xusb source, it seems that the problem arises in get_hid_input_record_size(), since the "found_direction" flag stays false and therefore record_size[j] stays zero.

Yes, I'm afraid that get_hid_input_record_size() was something I crafted very quickly so that it worked against the devices I had at my disposal, and which I had quite some trouble getting right actually. There doesn't seem to be any easy reference on how to deduce the report sizes from full fledged report descriptor (having to digest the whole report descriptor section from the HID specs to recreate a parser is more time than I was willing to allocate on this) and I didn't find any HID report descriptor parser code that I could simply copy/paste. Especially as this was only needed for non Windows environments, since we know exactly where exactly to lift our data in the fake Windows report, as little time as possible was spent on this section of code.

Interestingly, just notice that libusb xusb example has the following codes. I do not have the HID test device in xusb.c so I had never tested this. I dount it would work for Windows HID driver.

https://github.com/libusb/libusb/blob/master/examples/xusb.c#L616

	printf("\nReading HID Report Descriptors:\n");
	descriptor_size = libusb_control_transfer(handle, LIBUSB_ENDPOINT_IN|LIBUSB_REQUEST_TYPE_STANDARD|LIBUSB_RECIPIENT_INTERFACE,
		LIBUSB_REQUEST_GET_DESCRIPTOR, LIBUSB_DT_REPORT<<8, 0, hid_report_descriptor, sizeof(hid_report_descriptor), 1000);
	if (descriptor_size < 0) {
		printf("   Failed\n");
		return -1;
	}

There is a warning message generated under MSYS2.

C:\work\hid\hidapi_cmake\windows\hid.c: In function 'hid_enumerate':
C:\work\hid\hidapi_cmake\windows\hid.c:459:5: warning: 'strncpy' specified bound depends on the length of the source argument [-Wstringop-overflow=]
  459 |     strncpy(cur_dev->path, str, len+1);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

$ gcc -v
Using built-in specs.
COLLECT_GCC=C:\msys64\mingw64\bin\gcc.exe
COLLECT_LTO_WRAPPER=C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.3.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-10.3.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib --enable-bootstrap --enable-checking=release --with-arch=x86-64 --with-tune=generic --enable-languages=c,lto,c++,fortran,ada,objc,obj-c++,jit --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts=yes --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-lto --enable-libgomp --disable-multilib --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev1, Built by MSYS2 project' --with-bugurl=https://github.com/msys2/MINGW-packages/issues --with-gnu-as --with-gnu-ld --with-boot-ldflags='-pipe -Wl,--dynamicbase,--high-entropy-va,--nxcompat,--default-image-base-high -Wl,--disable-dynamicbase -static-libstdc++ -static-libgcc' 'LDFLAGS_FOR_TARGET=-pipe -Wl,--dynamicbase,--high-entropy-va,--nxcompat,--default-image-base-high' --enable-linker-plugin-flags='LDFLAGS=-static-libstdc++\ -static-libgcc\ -pipe\ -Wl,--dynamicbase,--high-entropy-va,--nxcompat,--default-image-base-high\ -Wl,--stack,12582912'
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 10.3.0 (Rev1, Built by MSYS2 project)

@Youw Yes the warning with windows\hid.c is there with Autotools as well. And yes it should be fixed in the source code.

Basically my idea is that a new API can do this.

1. under macOS and Linux, use native API
2. with libusb backend, use libusb API to dump the HID report descriptors
3. with Windows, the user needs to provide the raw data

pywinusb uses HidD_GetPhysicalDescriptor (the only routine I've found inside, that is closely related to HID descriptor) I believe someone tried to use this function with no luck so far

What about this? https://github.com/rene-aguirre/pywinusb/blob/master/pywinusb/hid/usage_pages.py

The HID Physical Descriptor is something different. It's a seldom used optional data structure of the HID protocol, which provides information about the specific part or parts of the human body that are activating a control. In case of a race car control, this would be "Both hands" for the steering wheel, and "Right hand" for the gear shift stick.

Traceback (most recent call last): File "raster2laser_gcode.py", line 37, in <module> import png File "C:\Program Files\Inkscape\share\inkscape\extensions\Inkscape-master\png.py", line 1885 except ValueError, e: ^ SyntaxError: invalid syntax

i am getting this error when trying to use gcode extension,i have no idea how to edit it since it is 2500 line of code

I'd say that hotplug support for each platform is pretty much straight-forward (I've done it for macOS and Windows several years ago in a propriatary project) same is true for linux (udev) and libusb backends

it just has to be done

pywinusb uses HidD_GetPhysicalDescriptor I believe someone tried to use this function with no luck so far

For Windows, not so sure pywinusb can shed some light. https://github.com/rene-aguirre/pywinusb

Or the codes here. http://janaxelson.com/hidpage.htm

I think Hidsharp also supports hotplug. https://github.com/IntergatedCircuits/HidSharp

Not so sure if we can get some ideas from here too. But it is using Windows API so probably it has the same limitation. https://github.com/rene-aguirre/pywinusb/tree/master/pywinusb/hid

@mcuee

I kind of know why it does not work for my device

I added that comment because I aware of that peculiarity of IOCTL_USB_GET_DESCRIPTOR_FROM_NODE_CONNECTION. It is documented here: https://docs.microsoft.com/windows-hardware/drivers/ddi/usbioctl/ns-usbioctl-_usb_descriptor_request But on my Windows 10 1909 PC it really works and I can dump HID Report Descriptors.

It may be that this Windows version just caching that descriptors (they are get requested by HIDClass driver after device connection). Because didn't saw these request via Wireshark while running my program. Just a speculation.

libusb backend is a bit different.

We haven't implemented the "split by usage_page/usage" for libusb yet.

BTW, I was puzzled why hidapi was not able to find my Microchip PICkit 2 programmer. Then I remember that many years ago (in May 2008) I submitted a trivial kernel patch (released in 2.6.26) to blacklist the device from kernel HID driver.

usbhid-dump does not care as it is based on libusb. mcuee@rpi400Ubuntu:~/build/libusb/hidapi$ usbhid-dump -d 04d8:0033 001:017:000:DESCRIPTOR 1624363237.908977 06 00 FF 09 01 A1 01 19 01 29 40 15 00 26 FF 00 75 08 95 40 81 02 19 01 29 40 91 02 C0

libusb backend is a bit different.

mcuee@rpi400Ubuntu:~/build/libusb/hidapi$ sudo ./hidtest/hidtest-libusb 
hidapi test/example tool. Compiled with hidapi version 0.10.1, runtime version 0.10.1.
Compile-time version matches runtime version of hidapi.

Device Found
  type: 04d9 0007
  path: 0001:0004:00
  serial_number: (null)
  Manufacturer:  
  Product:      Raspberry Pi Internal Keyboard
  Release:      161
  Interface:    0
  Usage (page): 0x0 (0x0)

Device Found
  type: 04d9 0007
  path: 0001:0004:01
  serial_number: (null)
  Manufacturer:  
  Product:      Raspberry Pi Internal Keyboard
  Release:      161
  Interface:    1
  Usage (page): 0x0 (0x0)

Device Found
  type: 1915 1025
  path: 0001:0007:02
  serial_number: (null)
  Manufacturer: ZY.Ltd
  Product:      ZY Control Mic
  Release:      173
  Interface:    2
  Usage (page): 0x0 (0x0)

Device Found
  type: 1915 1025
  path: 0001:0007:03
  serial_number: (null)
  Manufacturer: ZY.Ltd
  Product:      ZY Control Mic
  Release:      173
  Interface:    3
  Usage (page): 0x0 (0x0)

Device Found
  type: 046d c52b
  path: 0001:0005:00
  serial_number: (null)
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    0
  Usage (page): 0x0 (0x0)

Device Found
  type: 046d c52b
  path: 0001:0005:01
  serial_number: (null)
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    1
  Usage (page): 0x0 (0x0)

Device Found
  type: 046d c52b
  path: 0001:0005:02
  serial_number: (null)
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x0 (0x0)

unable to open device

And actually under Linux, I can see my USB Composite Device (Logitech Unifying Receiver) is also represented by multiple devices with hidapi hidraw backend.

mcuee@rpi400Ubuntu:~/build/libusb/hidapi$ ./hidtest/hidtest-hidraw 
hidapi test/example tool. Compiled with hidapi version 0.10.1, runtime version 0.10.1.
Compile-time version matches runtime version of hidapi.

Device Found
  type: 046d 101b
  path: /dev/hidraw5
  serial_number: 101b-ae-8a-0c-8c
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x2 (0x1)

Device Found
  type: 046d 101b
  path: /dev/hidraw5
  serial_number: 101b-ae-8a-0c-8c
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x1 (0x1)

Device Found
  type: 046d 101b
  path: /dev/hidraw5
  serial_number: 101b-ae-8a-0c-8c
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x1 (0xff00)

Device Found
  type: 046d 101b
  path: /dev/hidraw5
  serial_number: 101b-ae-8a-0c-8c
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x2 (0xff00)

Device Found
  type: 046d 101b
  path: /dev/hidraw5
  serial_number: 101b-ae-8a-0c-8c
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x4 (0xff00)

Device Found
  type: 046d 2008
  path: /dev/hidraw6
  serial_number: 2008-b6-b4-d3-96
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x6 (0x1)

Device Found
  type: 046d 2008
  path: /dev/hidraw6
  serial_number: 2008-b6-b4-d3-96
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x1 (0xc)

Device Found
  type: 046d 2008
  path: /dev/hidraw6
  serial_number: 2008-b6-b4-d3-96
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x80 (0x1)

Device Found
  type: 046d 2008
  path: /dev/hidraw6
  serial_number: 2008-b6-b4-d3-96
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x88 (0xffbc)

Device Found
  type: 046d 2008
  path: /dev/hidraw6
  serial_number: 2008-b6-b4-d3-96
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x1 (0xff00)

Device Found
  type: 046d 2008
  path: /dev/hidraw6
  serial_number: 2008-b6-b4-d3-96
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x2 (0xff00)

Device Found
  type: 046d 2008
  path: /dev/hidraw6
  serial_number: 2008-b6-b4-d3-96
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x4 (0xff00)

Device Found
  type: 046d c52b
  path: /dev/hidraw4
  serial_number: 
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x1 (0xff00)

Device Found
  type: 046d c52b
  path: /dev/hidraw4
  serial_number: 
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x2 (0xff00)

Device Found
  type: 046d c52b
  path: /dev/hidraw4
  serial_number: 
  Manufacturer: Logitech
  Product:      USB Receiver
  Release:      2410
  Interface:    2
  Usage (page): 0x4 (0xff00)

Device Found
  type: 1915 1025
  path: /dev/hidraw3
  serial_number: 
  Manufacturer: ZY.Ltd
  Product:      ZY Control Mic
  Release:      173
  Interface:    2
  Usage (page): 0x6 (0x1)

Device Found
  type: 1915 1025
  path: /dev/hidraw2
  serial_number: 
  Manufacturer: ZY.Ltd
  Product:      ZY Control Mic
  Release:      173
  Interface:    3
  Usage (page): 0x2 (0x1)

Device Found
  type: 1915 1025
  path: /dev/hidraw2
  serial_number: 
  Manufacturer: ZY.Ltd
  Product:      ZY Control Mic
  Release:      173
  Interface:    3
  Usage (page): 0x1 (0x1)

Device Found
  type: 1915 1025
  path: /dev/hidraw2
  serial_number: 
  Manufacturer: ZY.Ltd
  Product:      ZY Control Mic
  Release:      173
  Interface:    3
  Usage (page): 0x1 (0xc)

Device Found
  type: 1915 1025
  path: /dev/hidraw2
  serial_number: 
  Manufacturer: ZY.Ltd
  Product:      ZY Control Mic
  Release:      173
  Interface:    3
  Usage (page): 0x80 (0x1)

Device Found
  type: 1915 1025
  path: /dev/hidraw2
  serial_number: 
  Manufacturer: ZY.Ltd
  Product:      ZY Control Mic
  Release:      173
  Interface:    3
  Usage (page): 0x0 (0xff00)

Device Found
  type: 04d9 0007
  path: /dev/hidraw1
  serial_number: 
  Manufacturer:  
  Product:      Raspberry Pi Internal Keyboard
  Release:      161
  Interface:    0
  Usage (page): 0x6 (0x1)

Device Found
  type: 04d9 0007
  path: /dev/hidraw0
  serial_number: 
  Manufacturer:  
  Product:      Raspberry Pi Internal Keyboard
  Release:      161
  Interface:    1
  Usage (page): 0x1 (0xc)

unable to open device

From #249 comments from Youw .

We're doing quite the opposite for other platforms: we report same device multiple times, so each such unique device has its own usage_page/usage filled up.

And it kind of bits off a general design of HIDAPI - to be as simple as possible and just be a thin proxy over an OS-specific API/implementation, just enough to keep the uniform API across different platforms. (libusb backend is kind-of the most complicated in this matter, as it practically implements its own USB HID driver, but let it be the only exception.)

There is a warning under MSYS2 MinGW-w64.

I wounder if we have that warning when built with Autotools (if I had to guess - it is there too). In any case, we better fix the warning in the source code, i.e. I won't tweak the build system to hide the warning.

Looking at the VS2019 build, it seems to me the directory is a bit strange.

That is the default behavior of CMake - by default it generates under the build tree same directory structure as the source tree, and places each library/executable under the the directory, which defines it in the source.

In any case, by CMake ideology - no one should try to expect/get specific artifacts from whithin the build tree, but rather run install target and get everything you need from install diretrory (it doesn't have to be a /usr/local - it could be any diretory on the filesystem).

@DJm00n I looked at the code from https://github.com/DJm00n/RawInputDemo/tree/hid_report_descriptor_dump and I kind of know why it does not work for my device.

https://github.com/DJm00n/RawInputDemo/blob/hid_report_descriptor_dump/RawInputLib/RawInputDevice.cpp#L308

           // According to HID spec we need to do Report Descriptor request with
            // bmRequest set to 0x81 (Interface_In) but seems its also working with 0x80 for some reason

Apparently 0x80 does not work for my device.

/********************************************************************
  File Information:
    FileName:       usb_function_hid.c
    Dependencies:   See INCLUDES section
    Processor:      PIC18 or PIC24 USB Microcontrollers
    Hardware:       The code is natively intended to be used on the following
                    hardware platforms: PICDEM™ FS USB Demo Board, 
                    PIC18F87J50 FS USB Plug-In Module, or
                    Explorer 16 + PIC24 USB PIM.  The firmware may be
                    modified for use on other USB platforms by editing the
                    HardwareProfile.h file.
    Complier:  	    Microchip C18 (for PIC18) or C30 (for PIC24)
    Company:        Microchip Technology, Inc.
    
    Software License Agreement:
    
    The software supplied herewith by Microchip Technology Incorporated
    (the “Company”) for its PIC® Microcontroller is intended and
    supplied to you, the Company’s customer, for use solely and
    exclusively on Microchip PIC Microcontroller products. The
    software is owned by the Company and/or its supplier, and is
    protected under applicable copyright laws. All rights are reserved.
    Any use in violation of the foregoing restrictions may subject the
    user to criminal sanctions under applicable laws, as well as to
    civil liability for the breach of the terms and conditions of this
    license.
    
    THIS SOFTWARE IS PROVIDED IN AN “AS IS” CONDITION. NO WARRANTIES,
    WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING, BUT NOT LIMITED
    TO, IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
    PARTICULAR PURPOSE APPLY TO THIS SOFTWARE. THE COMPANY SHALL NOT,
    IN ANY CIRCUMSTANCES, BE LIABLE FOR SPECIAL, INCIDENTAL OR
    CONSEQUENTIAL DAMAGES, FOR ANY REASON WHATSOEVER.
    ...

void USBCheckHIDRequest(void)
{
    if(SetupPkt.Recipient != USB_SETUP_RECIPIENT_INTERFACE_BITFIELD) return;
    if(SetupPkt.bIntfID != HID_INTF_ID) return;
    
    /*
     * There are two standard requests that hid.c may support.
     * 1. GET_DSC(DSC_HID,DSC_RPT,DSC_PHY);
     * 2. SET_DSC(DSC_HID,DSC_RPT,DSC_PHY);
     */
    if(SetupPkt.bRequest == USB_REQUEST_GET_DESCRIPTOR)
    {
        switch(SetupPkt.bDescriptorType)
        {
            case DSC_HID: //HID Descriptor          
                if(USBActiveConfiguration == 1)
                {
                    USBEP0SendROMPtr(
                        (ROM BYTE*)&configDescriptor1 + 18,		//18 is a magic number.  It is the offset from start of the configuration descriptor to the start of the HID descriptor.
                        sizeof(USB_HID_DSC)+3,
                        USB_EP0_INCLUDE_ZERO);
                }
                break;
            case DSC_RPT:  //Report Descriptor           
                if(USBActiveConfiguration == 1)
                {
                    USBEP0SendROMPtr(
                        (ROM BYTE*)&hid_rpt01,
                        sizeof(hid_rpt01),     //See usbcfg.h
                        USB_EP0_INCLUDE_ZERO);
                }
                break;
            case DSC_PHY:  //Physical Descriptor
                //Note: The below placeholder code is commented out.  HID Physical Descriptors are optional and are not used
                //in many types of HID applications.  If an application does not have a physical descriptor,
                //then the device should return STALL in response to this request (stack will do this automatically
                //if no-one claims ownership of the control transfer).
                //If an application does implement a physical descriptor, then make sure to declare
                //hid_phy01 (rom structure containing the descriptor data), and hid_phy01 (the size of the descriptors in bytes),
                //and then uncomment the below code.
                //if(USBActiveConfiguration == 1)
                //{
                //    USBEP0SendROMPtr((ROM BYTE*)&hid_phy01, sizeof(hid_phy01), USB_EP0_INCLUDE_ZERO);
                //}
                break;
        }//end switch(SetupPkt.bDescriptorType)
    }//end if(SetupPkt.bRequest == GET_DSC)

This would use the source tree as the build directory.

Of course it wouldn't. It will create a build tree under current directory, which by cd <build dir> is any directory on the filesystem.

cmake -S . -B build

This creates a build directory inside the source tree, which by my experience is almost as bad as using source tree for build tree.<br> I'm using a software that scans all files under the source tree to index symbols (automatically) and if the build directory is under source tree - it scans it too, during the build as the build goes... So I prefer (and always recommend) to move build directorry somewhere outside the source tree completely.

@DJm00n Thanks for the info.

@mcuee regarding Chromium HID descriptor reconstruction code. You can look at real HIDP_PREPARSED_DATA in leaked Windows XP source code. See hidparse.h.

But keep in mind that exact struct could be changed since WinXp and that struct was hidden for a reason:

The internal structure of a _HIDP_PREPARSED_DATA structure is reserved for internal system use.

PS: Also there is code for kernel debugger command !hidkd.hidppd: https://github.com/cryptoAlgorithm/nt5src/blob/daad8a087a4e75422ec96b7911f1df4669989611/Source/XPSP1/NT/base/tools/kdexts2/hid.c#L223-L309