profile
viewpoint

neilpa/cmd-colors-solarized 1087

Solarized color settings for Windows command prompt

jspahrsummers/library-oriented-programming 58

Talk about the how/why of separating concerns into libraries

neilpa/circulate 56

Reverse engineered iOS library for controlling the Anova 2 via Bluetooth

jspahrsummers/correct-behavior-through-type-safety 15

Talk about using types to prove correctness

jspahrsummers/statelessness-of-the-union 5

Presentation for RACDC 2015

neilpa/dotfiles 4

More than just dotfiles

neilpa/ErrNo 3

Swift enum wrapper for C errno values

neilpa/euler 2

Solutions to problems from Project Euler

neilpa/List 2

Lists in Swift

neilpa/LMDB 2

LMDB.framework

issue closedneilpa/photohack

Recreating adjustments for RAW images

Currently, osxphotos cannot export edited versions of RAW images because Photos stores no jpeg version of the adjustments -- it only stores the .plist with the adjustment data. I wonder if there's enough info in the adjustment data returned by photohack to re-create the edited image (e.g. apply all the adjustments via CoreImage) to create a rendered jpeg? Certainly beyond my skills at the moment but I'm willing to try to learn enough Swift to experiment with this. Alternatively, photohack might be able to request the rendered edited photo via PhotoKit.

closed time in 18 days

RhetTbull

issue commentneilpa/photohack

Recreating adjustments for RAW images

Merged and tagged a new release with a pre-compiled binary. For now it's a simple export that dumps <UUID>.jpg files in the provided directory.

photohack path/to/some.photoslibrary jpegs path/to/export <UUID>...

Could make this more flexible for interaction with osxphotos if you want to go this route.

RhetTbull

comment created time in 18 days

created tagneilpa/photohack

tagv0.0.4

Hacking on the macOS Photos app *.photoslibrary

created time in 18 days

release neilpa/photohack

v0.0.4

photohack 59.25KB

released time in 18 days

push eventneilpa/photohack

Neil Pankey

commit sha 352f0d5f6075bbcc70df01a6a16ecbc90e9b371b

jpegs: Command for exporting adjusted JPEGs

view details

Neil Pankey

commit sha ff8474aa059bab8d5faf8ed47f0f32bf10cdd291

readme: Update for JPEG export

view details

Neil Pankey

commit sha f2fa6cc6c4b2b2216cd99854e6e3336844371265

jpegs: version 0.0.4

view details

Neil Pankey

commit sha 1283e3936f6a57db267a437ab744808a2c67d203

Merge pull request #3 from neilpa/export-jpegs jpegs: Command for exporting adjusted JPEGs

view details

push time in 18 days

PR merged neilpa/photohack

jpegs: Command for exporting adjusted JPEGs

Export one or more photos as JPEGs with adjustments applied to a specified directory.

photohack path/to/some.photoslibrary jpegs path/to/export <UUID>...

Where each exported photo will be named <UUID>.jpg.

+112 -41

0 comment

5 changed files

neilpa

pr closed time in 18 days

push eventneilpa/photohack

Neil Pankey

commit sha f2fa6cc6c4b2b2216cd99854e6e3336844371265

jpegs: version 0.0.4

view details

push time in 18 days

push eventneilpa/photohack

Neil Pankey

commit sha ff8474aa059bab8d5faf8ed47f0f32bf10cdd291

readme: Update for JPEG export

view details

push time in 18 days

issue commentRhetTbull/osxphotos

Option for converting raw images on export

GitHub Actions which use virtual machines w/o GPUs.

It should be possible to detect this (e.g. the MTLDevice creation will fail) and fallback to a CPU-based context.

neilpa

comment created time in 18 days

issue commentneilpa/photohack

Recreating adjustments for RAW images

See #3 for a quick hack at exporting adjusted JPEGs.

RhetTbull

comment created time in 18 days

PR opened neilpa/photohack

jpegs: Command for exporting adjusted JPEGs

Export one or more photos as JPEGs with adjustments applied to a specified directory.

photohack jpegs path/to/dir <UUID>...

Where each exported photo will be named <UUID>.jpg.

+89 -29

0 comment

3 changed files

pr created time in 18 days

create barnchneilpa/photohack

branch : export-jpegs

created branch time in 18 days

issue commentneilpa/photohack

Recreating adjustments for RAW images

Found the relevant PhotoKit docs. The requestImage* methods of PHImageManager have an PHImageRequestOptions argument with a version property you can set to PHImageRequestOptionsVersion.current.

RhetTbull

comment created time in 18 days

issue commentneilpa/yajsv

UTF-8 with BOM and UTF-16 LE encoded files produce read error

@GlenAppleton I posted a v1.4.0 release with support for UTF-16 files. I also added the -b flag for allowing BOMs in JSON files where it would otherwise be an error.

Let me know if you have any issues. I haven't tried this manually on Windows but the large test matrix I added passes the Github CI build on Windows.

Have you considered adding a "debug" switch like -d, where you would send useful information to standard out, such as the text encoding?

I decided to not do this for now but made a note about the possibility in #12

GlenAppleton

comment created time in 19 days

issue commentneilpa/yajsv

Add flag for zero-output/exit code only

See also https://github.com/neilpa/yajsv/issues/13#issuecomment-694932174 for adding additional output such as the text encoding in a verbose mode.

gabesullice

comment created time in 19 days

created tagneilpa/yajsv

tagv1.4.0

Yet Another JSON Schema Validator [CLI]

created time in 19 days

release neilpa/yajsv

v1.4.0

released time in 19 days

delete branch neilpa/yajsv

delete branch : unicode-encodings

delete time in 19 days

push eventneilpa/yajsv

Neil Pankey

commit sha 6077a84cc2aee7a73dd501f80c4d54b4b72d9901

test: build on go 1.11

view details

Neil Pankey

commit sha 6f93deba88eba7463860d4c854aa7ac5f08e9140

test: generate utf-16 testdata and bom variants

view details

Neil Pankey

commit sha 806c9cd19a24e27e91c31a1f56f140cbcae32c02

yajsv: UTF-16 and BOM handling

view details

Neil Pankey

commit sha 87a159c4587a194fa9386fb340980fec53bd61cf

utf16: Schema errors and encoding tests

view details

Neil Pankey

commit sha b2b9a856775c2bb0f3363344aaa286b9b39a8209

Merge pull request #14 from neilpa/unicode-encodings UTF-16 support for JSON and option to skip BOMs

view details

push time in 19 days

PR merged neilpa/yajsv

UTF-16 support for JSON and option to skip BOMs

Adds support to detect and decode UTF-16 (BE or LE) JSON text. Also adds an option to skip BOMs at the start of a JSON document. These are treated as errors by default (with a better message) based on RFC 8259. Resolves #13

TODO

  • [x] Tests for the data generated in 6f93deb for the new encoding combos
  • [ ] Tests for one and two byte JSON docs
+286 -44

0 comment

51 changed files

neilpa

pr closed time in 19 days

issue closedneilpa/yajsv

UTF-8 with BOM and UTF-16 LE encoded files produce read error

I've been running some tests using your yajsv (windows 64 version) utility, and I've found that both UTF-8 (BOM) and UTF-16 LE encoded files throw the following errors:

UTF-8 with BOM : error: validate: invalid character 'ï' looking for beginning of value UTF-16 LE : error: validate: invalid character 'ÿ' looking for beginning of value

For my purpose implementing in a PowerShell script, it's not a big deal since I can re-encode the files to standard UTF-8 and it works, but I thought you'd like to know about the encoding limitations.

Great utility, BTW!

closed time in 19 days

GlenAppleton

push eventneilpa/yajsv

Neil Pankey

commit sha 87a159c4587a194fa9386fb340980fec53bd61cf

utf16: Schema errors and encoding tests

view details

push time in 19 days

issue commentneilpa/photohack

Recreating adjustments for RAW images

Exporting the rendered edit via PhotoKit would be ideal. I'm pretty sure I saw an option for that as part of fetching asset image data but now I'm struggling to find it again in the PhotoKit docs.

If that doesn't work, it should be possible to apply all the adjustments now that I've figured out the schema and how to extract the associated dictionaries. There's a lot of them though so it's going to be non-trivial to map them all onto onto CoreImage operations.

RhetTbull

comment created time in 20 days

issue commentRhetTbull/osxphotos

edits to raw images are not exported

I checked and the edited JPEGs exist on Catalina under resources/renders/${UUID_PREFIX}/${UUID}_1_201_a.jpeg. (Didn't check the DB though to see if that's referenced somewhere).

RhetTbull

comment created time in 20 days

issue commentRhetTbull/osxphotos

Export modifications as well?

Could re-open this now based on the progress in https://github.com/RhetTbull/osxphotos/issues/215#issuecomment-698663130. Would also keep that issue focused on RAW conversion and could further discuss how to handle adjustment data here.

mccarverone

comment created time in a month

issue commentRhetTbull/osxphotos

Option for converting raw images on export

I did more digging into the adjustment data and seem to have figured out the schema for the full scope of edits that are possible in the Photos app. There's a lot of details in https://github.com/neilpa/photohack/pull/1. I haven't done much testing yet on sample images but would be interesting to run this through the test libraries in this repo.

I also figured out how to make a standalone binary work so I attached that to a release. That should make it a bit easier to experiment with. For now I just have it dump a JSON map with the asset UUID as keys and the adjustment dictionaries as values.

Or, the interface could be a local webserver with RESTful API though that would take more work than a simple CLI.

That would be an interesting approach but I'm not sure how simple it is to do something like this in Swift/Objective-C out of the box. I usually default to Go for CLI apps for this reason, it's trivial to add a web/http interface if needed. In this case I didn't want to add yet another language into the mix.

neilpa

comment created time in a month

created tagneilpa/photohack

tagv0.0.3

Hacking on the macOS Photos app *.photoslibrary

created time in a month

release neilpa/photohack

v0.0.3

released time in a month

delete branch neilpa/photohack

delete branch : adjustments-json

delete time in a month

push eventneilpa/photohack

Neil Pankey

commit sha 183e795c57834ba11ba73d0c379356a7cb764139

adjustments: reverse engineered the schema

view details

Neil Pankey

commit sha d62e4c23cbb891b2c9c8ae9f8e4c03024d8df31f

adjustments: JSON-ify the schema

view details

Neil Pankey

commit sha 3f7f168716d8a9d01beb025b949d67cea54ba441

adjustments: JSON-ify the asset output

view details

Neil Pankey

commit sha 5655455d2642e43660eb8cec6f6d25f7f2c0193d

photohack: v0.0.3

view details

Neil Pankey

commit sha e15392c41767e4375c52b5447840783b386897ff

Merge pull request #1 from neilpa/adjustments-json Decode adjustments as JSON

view details

push time in a month

PR merged neilpa/photohack

Decode adjustments as JSON

The README contains most of the details. This decodes the adjustment data as a JSON map for a given set of asset UUIDs.

Additionally there's notes, links, debug spew, and helper function that extracted the entire array of ajdustment possibilities in the Photos app. Note the "array" and "compound" adjustments may not be right as those are more complicated and haven't been tested yet.

+2557 -76

0 comment

7 changed files

neilpa

pr closed time in a month

PR opened neilpa/photohack

Decode adjustments as JSON

The README contains most of the details. This decodes the adjustment data as a JSON map for a given set of asset UUIDs.

Additionally there's notes, links, debug spew, and helper function that extracted the entire array of ajdustment possibilities in teh Photos app. Note the "array" and "compound" adjustments may not be right as those are more complicated and haven't been tested yet.

+2557 -76

0 comment

7 changed files

pr created time in a month

create barnchneilpa/photohack

branch : adjustments-json

created branch time in a month

push eventneilpa/photohack

Neil Pankey

commit sha c988c576b1dc92ef6f457f72daf29864e532a6bd

photohack: embed and extract Info.plist Works around the OS not checking the embedded plist so that photohack can request permissions for photo library access without crashing.

view details

push time in a month

startedneilpa/photohack

started time in a month

issue commentRhetTbull/osxphotos

Option for converting raw images on export

Here's my initial Poc Xcode project - https://github.com/neilpa/photohack

If you add a new Xcode scheme with the first argument as a path to a *.photoslibrary and subsequent arguments as asset UUIDs it'll dump adjustment data in a readable format. That should get you something like the following.

{
    Album = PHAssetCollection;
    Asset = PHAsset;
    CloudSharedAlbum = PHCloudSharedAlbum;
    DetectedFace = PHFace;
    DetectedFaceGroup = PHFaceGroup;
    FaceCrop = PHFaceCrop;
    FetchingAlbum = PHAssetCollection;
    Folder = PHCollectionList;
    GenericAsset = PHAsset;
    ImportSession = PHImportSession;
    Keyword = PHKeyword;
    LegacyFaceAlbum = PHAssetCollection;
    Memory = PHMemory;
    Moment = PHMoment;
    MomentList = PHMomentList;
    MomentShare = PHMomentShare;
    MomentShareParticipant = PHMomentShareParticipant;
    Person = PHPerson;
    PhotoStreamAlbum = PHAssetCollection;
    PhotosHighlight = PHPhotosHighlight;
    ProjectAlbum = PHProject;
    Question = PHQuestion;
    Suggestion = PHSuggestion;
}
adjustments: E1C9A934-260F-4211-95AD-14827A82ACB4
  <none>
2020-09-22 20:17:59.046444-0700 photohack[13032:3222620] Metal API Validation Enabled
adjustments: 34B8483F-8A35-4FEB-98A3-84DC376F0C9F
  composition = <NUGenericComposition:0x100457b60 id=me.neilpa.photohack:PhotosComposition~1.0 mediaType=Unknown contents={
	raw = <NUGenericAdjustment:0x100458080> id=me.neilpa.photohack:RAW~1.0 settings={
    auto = 0;
    enabled = 1;
    inputDecoderVersion = 8;
},
	cropStraighten = <NUGenericAdjustment:0x1004582e0> id=me.neilpa.photohack:CropStraighten~1.0 settings={
    angle = "-0";
    auto = 0;
    constraintHeight = 0;
    constraintWidth = 0;
    enabled = 1;
    height = 988;
    width = 1482;
    xOrigin = 2342;
    yOrigin = 1127;
},
	orientation = <NUGenericAdjustment:0x100458610> id=me.neilpa.photohack:Orientation~1.0 settings={
    value = 1;
},
}>

I'll try to get a proper binary built (and updated instructions) but haven't figured out how to properly embed the Info.plist such that it can request photos library permissions.

neilpa

comment created time in a month

create barnchneilpa/photohack

branch : master

created branch time in a month

created repositoryneilpa/photohack

Hacking on the macOS Photos app *.photoslibrary

created time in a month

issue commentRhetTbull/osxphotos

Handling of RAW images needs work

There's also https://raw.pixls.us/ which is the successor to https://rawsamples.ch

RhetTbull

comment created time in a month

issue commentRhetTbull/osxphotos

Option for converting raw images on export

Yea, you definitely want to reuse the context if possible. (There was a similar comment the the SO post I linked).

I think PhotoKit will only work with the "system library" and many people have more than one Photos library.

After spending some time with the private headers I was able to hack together some Objective-C code that successfully loads a photos library from an arbitrary path. I got far enough to fetch a PHAsset via UUID and validate a few properties, including the path on disk is expected.

I need to clean this up a bit but I'll push a proof-of-concept Xcode project and share more of my findings. There's some interesting stuff in the header dump that I want to look at more. In particular, I think I've found a way to extract the adjustment data (at least the crop region along with orientation and rotation).

neilpa

comment created time in a month

issue commentRhetTbull/osxphotos

Option for converting raw images on export

Good call on CoreImage, that makes the conversion nearly trivial. I did a quick test with that approach and it worked great for the raw images I tried.

As for the PhotoKit limitation, it may be possible to use a custom library via the private header initialization methods. I'm one of those folks with a non-standard library and will see if I can get something working here.

neilpa

comment created time in a month

issue commentRhetTbull/osxphotos

Option for converting raw images on export

I hadn't actually used dcraw before and was doing some experiments this morning. I'm not sure it's going to work as well as I had hoped, at least for bulk exports. The default color correction leaves a lot to be desired. I also experimented with various options and got slightly better results. However, it doesn't compare to the default results I get from the Photos app (or something like Darktable). I'm very much a noob at "developing" digital photos so I may be overlooking something obvious.

There's also darktable-cli which I want to test. Maybe that will work better out of the box.

I'm not competent in Swift or Objective-C

It's been awhile, but I've used both a fair bit. I may experiment this afternoon to see what's possible in terms of export.

neilpa

comment created time in a month

starteddarktable-org/rawspeed

started time in a month

startedRhetTbull/osxphotos

started time in a month

issue commentRhetTbull/osxphotos

Option for converting raw images on export

This is the answer I was afraid of.

For .heic, /usr/bin/sips can do the conversion

Interesting, I wasn't aware of this built-in command and had been using the libheif tools

Photos itself can convert the images.

Yea, I had a small hope that this would be available via PhotoKit without having to script the GUI. I can't imagine going that route once you start getting above 1k photos. Exporting of my full library of ~16k photos to a fast external SSD and that takes ~20 minutes currently. Seems unlikely that would go well when invoking Photos for each image given the occassional crashes I already see.

Another possibility would be to add an option to allow the user to run a command of choice

I suspect this would be hard to make work in practice. You'll likely need different commands for heif (e.g. sips) vs raw (e.g. dcraw) files. You could wrap it in a shell script, but then you lose the benefits of the templating system. Maybe there's an ergonomic way to do this on an per-extension (or UTI?) basis. However, I worry the plethora of raw file formats would make this impractical.

the resulting converted photo would not be tracked by osxphotos so it would not be considered when running export --updated

I haven't looked into the mechanics of how --updated works yet. My current thinking (sans builtin conversion) is to do an export, do a post-hoc conversion, then truncate the RAW files to 0 bytes keeping the exported file created/mod times. Would that "trick" osxphotos into not re-exporting images unless they had actually been updated in the Photos app?

neilpa

comment created time in a month

issue openedRhetTbull/osxphotos

Option for converting raw images on export

Would it be feasible to add an option to always convert RAW images on export to JPEG? Any edited photos are already going to be JPEG but I've got a lot of un-edited raws w/out the JPEG preview. This would allow something like the following (totally made up --raw-to-jpeg) to quickly create a backup that is also extremely portable.

osxphotos export --raw-to-jpeg --export-by-date \
  --skip-original-if-edited --edit-suffix "" \
  --no-extended-attributes /Volumes/thumbdrive

created time in a month

PR opened neilpa/yajsv

UTF-16 support for JSON and option to skip BOMs

Adds support to detect and decode UTF-16 (BE or LE) JSON text. Also adds an option to skip BOMs at the start of a JSON document. These are treated as errors by default (with a better message) based on RFC 8259. Resolves #13

TODO

  • [ ] Tests for the data generated in 6f93deb for the new encoding combos
  • [ ] Tests for one and two byte JSON docs
+179 -34

0 comment

51 changed files

pr created time in a month

push eventneilpa/yajsv

Neil Pankey

commit sha 806c9cd19a24e27e91c31a1f56f140cbcae32c02

yajsv: UTF-16 and BOM handling

view details

push time in a month

create barnchneilpa/yajsv

branch : unicode-encodings

created branch time in a month

issue commentneilpa/yajsv

UTF-8 with BOM and UTF-16 LE encoded files produce read error

Thanks for looking more closely at the RFCs. Obviously I didn't read those carefully enough last night. Given that second sentence it seems reasonable to allow it.

you could also consider it a warning condition and maybe add a switch to ignore warnings, leaving the default to treat them as errors.

Right now files are bucketed to a pass, fail, or error, the last being invalid JSON. I'd say these should still default to an "error" but with a better message about the BOM. The rationale being the moment they are served as-is over the internet the become invalid.

As for the switch, I'm thinking along the lines of -allow-bom to opt-out of that behavior. Although that would diverge from my current single character flags for everything. Might be time to rethink/break given the feedback in #12

GlenAppleton

comment created time in a month

issue commentneilpa/yajsv

UTF-8 with BOM and UTF-16 LE encoded files produce read error

Further complicating matters, YAML does support BOMs and required for UTF-16 detection.

GlenAppleton

comment created time in a month

issue commentneilpa/yajsv

UTF-8 with BOM and UTF-16 LE encoded files produce read error

Turns out it's invalid to include a BOM in a JSON document. I'll make this case fail with a better error message but it doesn't make sense for a JSON validator to accept invalid JSON :wink:. As for the other UTF-* encodings, that post offers a simple approach to detect them based on the first few bytes.

GlenAppleton

comment created time in a month

issue commentRhetTbull/osxphotos

Export: / character in album name not handled properly when album name used as directory

In Catalina, Finder does show the "/" in place of the ":"

Woah, crazy that Finder blindly replaces : in a directory name with / for display. Guess this originates from the pre-OSX days which was before I switched to macs.

garz75

comment created time in a month

issue commentneilpa/yajsv

UTF-8 with BOM and UTF-16 LE encoded files produce read error

I had forgotten about the prevalence of UTF-16 on Windows although I can't say I've ever seen UTF-8 files with a BOM. I suspect this would be relatively easy to detect and support via the golang.org/x/text package.

Out of curiosity, do you happen to know what utilities generated the UTF-8 w/ BOM files?

GlenAppleton

comment created time in a month

startedsecretGeek/csvz

started time in 2 months

issue commentRhetTbull/osxphotos

XMP sidecar collisions

Both seems like the right approach to me. I know that darktable defaults to appending .xmp while keeping the original file extension.

mccarverone

comment created time in 2 months

issue commentRhetTbull/osxphotos

Feature: provide interface to the detailed faces information in database

I've not been able to find a way to test this. I've tried both Digikam and Lightroom, both of which I'm unfamiliar with, and can't figure out how to get them to import face regions from an XMP file despite hours of googling

I hit this exact same issue when working on this in phace. I could not find an app that actually used the encoded face regions. IIRC the docs for DigiKam claimed to but in all my experimenting I couldn't find an example of it working. Best I did was find some sample images with the XMP face regions already encoded and made sure I could round-trip the data based on my understanding of the format.

RhetTbull

comment created time in 3 months

more