profile
viewpoint

issue closed0xThiebaut/virustotal

Github Action Workflow

Would it be possible to use this same method to check known virus total hashes in github action workflow?

closed time in 4 days

SteveBox0

issue opened0xThiebaut/virustotal

Github Action Workflow

Would it be possible to use this same method to check known virus total hashes in github action workflow?

created time in 22 days

issue openedcrazy-max/ghaction-virustotal

Option to use cached hash results

I would like to see an option to use cached VT results if the hash already exists. This way keeping the API limit for files that do not exists on VT already. This idea is similar to the below project.

https://github.com/0xThiebaut/virustotal

created time in 24 days

issue openediamibrahim-dev/virustotalscanner

Update to V3 API and support larger files

https://developers.virustotal.com/reference/files-upload-url

created time in 2 months

issue openeddiogo-fernan/malsub

Add support for uploading larger files

https://developers.virustotal.com/reference/files-upload-url

created time in 2 months

issue openedalexandreborges/malwoverview

Add support for uploading larger files

E.g. https://developers.virustotal.com/reference/files-upload-url

created time in 2 months

issue openedsgtdede/multiscanners-upload

Fails to upload 51MB zip file

From the looks your current method of uploading does not support larger files. VT can support uploading large files.

Traceback (most recent call last):
  File "/home/user/Documents/3/multiscanners-upload/uploader.py", line 255, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/user/Documents/3/multiscanners-upload/uploader.py", line 41, in main
    await asyncio.gather(
  File "/home/user/Documents/3/multiscanners-upload/uploader.py", line 60, in upload_virustotal
    text = await resp.json()
  File "/home/user/.local/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 1103, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8', url=URL('https://www.virustotal.com/api/v3/files')

created time in 2 months

issue openedb4tman/docker-squid

assertion failed: tunnel.cc:1227: "!waitingForConnectExchange"

Under load the squid container seems to restart the process very often and drop all current connections.

kid1| assertion failed: tunnel.cc:1227: "!waitingForConnectExchange"
    current master transaction: master55

created time in 2 months

issue openedmarkus-wa/demoinfocs-golang

README.md invalid Discord invite link

The discord invite link included in the README.md is invalid. Could you please look at updating the readme.

https://discord.gg/69Y2MM7E

created time in 3 months

issue closedAlpha1337k/csgo-democheck

Contact Method?

Hi,

What is the best way to contact you? Do you have discord by any chance?

closed time in 3 months

SteveBox0

issue openedAlpha1337k/csgo-democheck

Contact Method?

Hi,

What is the best way to contact you? Do you have discord by any chance?

created time in 3 months

issue commentrclone/rclone

Windows sha1sum via SFTP

Using SHA1 and around 1000 small files the Window server kernel time is over 50% during the test.

Which shell? Default/cmd, with sha1sum_command configured? Or PowerShell, with the built in script sum commands, or something else configured?

Default shell with --sftp-sha1sum-command "sha1sum"

SteveBox0

comment created time in 3 months

issue commentrclone/rclone

Windows sha1sum via SFTP

Also, tested rclone v1.57.0-beta.5806.a238877ad and rclone-v1.57.0-beta.5750.a3d16b4f8 with hasher backend. Should i create a new issue for this or is it related?

rclone cat "Hasher:/C:/test folder/" -vv 
<7>DEBUG : rclone: Version "v1.57.0-beta.5806.a238877ad" starting with parameters ["rclone" "cat" "Hasher:/C:/test folder/" "-vv"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "Hasher:/C:/test folder/"
<7>DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
<6>INFO  : Hasher is EXPERIMENTAL!
<7>DEBUG : Creating backend with remote "windows/C:/test folder"
<7>DEBUG : fs cache: renaming cache item "windows/C:/test folder" to be canonical "/home/user/Documents/windows/C:/test folder"
<7>DEBUG : hasher::Hasher:/C:/test folder/: Groups by usage: cached [sha1], passed [], auto [], slow [md5, sha1, whirlpool, crc32, sha256, dropbox, mailru, quickxor], supported [md5, sha1, whirlpool, crc32, sha256, dropbox, mailru, quickxor]
<3>ERROR : : error listing: directory not found
<7>DEBUG : 3 go routines active
Failed to cat with 2 errors: last error was: directory not found
[Hasher]
type = hasher
remote = windows
hashes = sha1
max_age = 1d
SteveBox0

comment created time in 3 months

issue commentrclone/rclone

Windows sha1sum via SFTP

The build (rclone-v1.57.0-beta.5809.af17cbe72.sftp-windows-server-linux-amd64) seems to be working fine but with one odd side effect. Using SHA1 and around 1000 small files the Window server kernel time is over 50% during the test.

SteveBox0

comment created time in 3 months

issue commentrclone/rclone

Windows sha1sum via SFTP

Please let me know if you any extra information for the bug fix.

SteveBox0

comment created time in 3 months

issue commentrclone/rclone

Windows sha1sum via SFTP

OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2 is the version currently included by Microsoft for both Server 2019 and Server 2022. Tested is couple of versions but the same issue. I dont think its related to the version but as you mentioned the shellEscape method.

SteveBox0

comment created time in 3 months

issue commentrclone/rclone

Windows sha1sum via SFTP

  • Direct Folder Test
rclone sha1sum "windows:/C:/test_folder/example1/" --sftp-sha1sum-command "sha1sum" -vv --sftp-path-override "C:\test_folder\example1"
2021/10/26 23:06:32 DEBUG : rclone: Version "v1.57.0-beta.5801.bb11803f1" starting with parameters ["rclone" "sha1sum" "windows:/C:/test_folder/example1/" "--sftp-sha1sum-command" "sha1sum" "-vv" "--sftp-path-override" "C:\\test_folder\\example1"]
2021/10/26 23:06:32 DEBUG : Creating backend with remote "windows:/C:/test_folder/example1/"
2021/10/26 23:06:32 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2021/10/26 23:06:32 DEBUG : windows: detected overridden config - adding "{zmpfa}" suffix to name
2021/10/26 23:06:32 DEBUG : sftp://****@****:22//C:/test_folder/example1/: New connection ****:57362->****:22 to "SSH-2.0-OpenSSH_for_Windows_8.1"
2021/10/26 23:06:32 DEBUG : fs cache: renaming cache item "windows:/C:/test_folder/example1/" to be canonical "windows{zmpfa}:/C:/test_folder/example1/"
2021/10/26 23:06:33 DEBUG : sftp cmd = C:\\test_folder\\example1/file1.txt
2021/10/26 23:06:33 DEBUG : sftp output = "dad6647784b11394806ca5f5ccea1d246b8b7a26  C:\\\\test_folder\\\\example1/file1.txt\r\n"
2021/10/26 23:06:33 DEBUG : sftp hash = "dad6647784b11394806ca5f5ccea1d246b8b7a26"
dad6647784b11394806ca5f5ccea1d246b8b7a26  file1.txt
2021/10/26 23:06:33 DEBUG : 11 go routines active
user@user-virtual-machine:~$ 
  • Sub Folder test
rclone sha1sum "windows:/C:/test_folder/" --sftp-sha1sum-command "sha1sum" -vv --sftp-path-override "C:\test_folder"
2021/10/26 23:07:57 DEBUG : rclone: Version "v1.57.0-beta.5801.bb11803f1" starting with parameters ["rclone" "sha1sum" "windows:/C:/test_folder/" "--sftp-sha1sum-command" "sha1sum" "-vv" "--sftp-path-override" "C:\\test_folder"]
2021/10/26 23:07:57 DEBUG : Creating backend with remote "windows:/C:/test_folder/"
2021/10/26 23:07:57 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2021/10/26 23:07:57 DEBUG : windows: detected overridden config - adding "{NNAPN}" suffix to name
2021/10/26 23:07:58 DEBUG : sftp://****@****:22//C:/test_folder/: New connection ****:57368->****:22 to "SSH-2.0-OpenSSH_for_Windows_8.1"
2021/10/26 23:07:58 DEBUG : fs cache: renaming cache item "windows:/C:/test_folder/" to be canonical "windows{NNAPN}:/C:/test_folder/"
2021/10/26 23:07:58 DEBUG : sftp cmd = C:\\test_folder/example1/file1.txt
2021/10/26 23:07:58 DEBUG : sftp output = "dad6647784b11394806ca5f5ccea1d246b8b7a26  C:\\\\test_folder/example1/file1.txt\r\n"
2021/10/26 23:07:58 DEBUG : sftp hash = "dad6647784b11394806ca5f5ccea1d246b8b7a26"
dad6647784b11394806ca5f5ccea1d246b8b7a26  example1/file1.txt
2021/10/26 23:07:58 DEBUG : 11 go routines active

-Space in Folder

rclone sha1sum "windows:/C:/test folder/" --sftp-sha1sum-command "sha1sum" -vv --sftp-path-override "C:\test folder"
2021/10/26 23:09:27 DEBUG : rclone: Version "v1.57.0-beta.5801.bb11803f1" starting with parameters ["rclone" "sha1sum" "windows:/C:/test folder/" "--sftp-sha1sum-command" "sha1sum" "-vv" "--sftp-path-override" "C:\\test folder"]
2021/10/26 23:09:27 DEBUG : Creating backend with remote "windows:/C:/test folder/"
2021/10/26 23:09:27 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2021/10/26 23:09:27 DEBUG : windows: detected overridden config - adding "{r5Yfc}" suffix to name
2021/10/26 23:09:27 DEBUG : sftp://****@****:22//C:/test folder/: New connection ****:57370->****:22 to "SSH-2.0-OpenSSH_for_Windows_8.1"
2021/10/26 23:09:27 DEBUG : fs cache: renaming cache item "windows:/C:/test folder/" to be canonical "windows{r5Yfc}:/C:/test folder/"
2021/10/26 23:09:27 DEBUG : sftp cmd = C:\\test\ folder/example1/file1.txt
2021/10/26 23:09:27 DEBUG : example1/file1.txt: Failed to calculate sha1 hash: Process exited with status 1 (can't open `C:\\test\': No such file or directory)
                                          example1/file1.txt
2021/10/26 23:09:27 DEBUG : 11 go routines active
SteveBox0

comment created time in 3 months

issue commentrclone/rclone

Windows sha1sum via SFTP

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/windows-sha1sum-via-sftp/27160

SteveBox0

comment created time in 3 months

issue openedrclone/rclone

Windows sha1sum via SFTP

What is the problem you are having with rclone?

I am using the official OpenSSH that is now part of windows (link below). This works well for file transfer but hash checks do not seem to work. https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse

I have tested using both sha1sum and md5sums.

https://www.adaic.org/resources/add_content/standards/sha1sum.exe --sftp-sha1sum-command "sha1sum"

http://www.pc-tools.net/files/win32/freeware/md5sums-1.2.zip --sftp-md5sum-command "md5sums -u"

The first issue seems to be the path being different to sftp and i got around this using --sftp-path-override "C:\test folder".

Looking at the console output the issue seems to be the way rclone handles the escape characters and backslashes.

I think the best way forward would most likely just be an OS detection on SSH connect or a optional flag for Windows.

What is your rclone version (output from rclone version)

v1.57.0-beta.5801.bb11803f1

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Client - Ubuntu 21.10 Server - Windows Server 2019

Which cloud storage system are you using? (e.g. Google Drive)

SFTP

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone sha1sum "windows:/C:/test folder/example1/" --sftp-sha1sum-command "sha1sum" -vv

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

2021/10/26 18:08:39 DEBUG : rclone: Version "v1.57.0-beta.5801.bb11803f1" starting with parameters ["rclone" "sha1sum" "windows:/C:/test folder/example1/" "--sftp-sha1sum-command" "sha1sum" "-vv"]
2021/10/26 18:08:39 DEBUG : Creating backend with remote "windows:/C:/test folder/example1/"
2021/10/26 18:08:39 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2021/10/26 18:08:39 DEBUG : windows: detected overridden config - adding "{h5EBD}" suffix to name
2021/10/26 18:08:40 DEBUG : sftp://*****@*****//C:/test folder/example1/: New connection *****:39518->*****:22 to "SSH-2.0-OpenSSH_for_Windows_8.1"
2021/10/26 18:08:40 DEBUG : fs cache: renaming cache item "windows:/C:/test folder/example1/" to be canonical "windows{h5EBD}:/C:/test folder/example1/"
2021/10/26 18:08:40 DEBUG : sftp cmd = /C:/test\ folder/example1/file1.txt
2021/10/26 18:08:40 DEBUG : file1.txt: Failed to calculate sha1 hash: Process exited with status 1 (can't open `/C:/test\': Invalid argument)
                                          file1.txt
2021/10/26 18:08:40 DEBUG : 11 go routines active

created time in 3 months

more