Currently, osxphotos cannot export edited versions of RAW images because Photos stores no jpeg version of the adjustments -- it only stores the .plist with the adjustment data. I wonder if there's enough info in the adjustment data returned by photohack to re-create the edited image (e.g. apply all the adjustments via CoreImage) to create a rendered jpeg? Certainly beyond my skills at the moment but I'm willing to try to learn enough Swift to experiment with this. Alternatively, photohack might be able to request the rendered edited photo via PhotoKit.

Merged and tagged a new release with a pre-compiled binary. For now it's a simple export that dumps <UUID>.jpg files in the provided directory.

photohack path/to/some.photoslibrary jpegs path/to/export <UUID>...

Could make this more flexible for interaction with osxphotos if you want to go this route.

Export one or more photos as JPEGs with adjustments applied to a specified directory.

photohack path/to/some.photoslibrary jpegs path/to/export <UUID>...

Where each exported photo will be named <UUID>.jpg.

GitHub Actions which use virtual machines w/o GPUs.

It should be possible to detect this (e.g. the MTLDevice creation will fail) and fallback to a CPU-based context.

See #3 for a quick hack at exporting adjusted JPEGs.

Export one or more photos as JPEGs with adjustments applied to a specified directory.

photohack jpegs path/to/dir <UUID>...

Where each exported photo will be named <UUID>.jpg.

Found the relevant PhotoKit docs. The requestImage* methods of PHImageManager have an PHImageRequestOptions argument with a version property you can set to PHImageRequestOptionsVersion.current.

@GlenAppleton I posted a v1.4.0 release with support for UTF-16 files. I also added the -b flag for allowing BOMs in JSON files where it would otherwise be an error.

Let me know if you have any issues. I haven't tried this manually on Windows but the large test matrix I added passes the Github CI build on Windows.

Have you considered adding a "debug" switch like -d, where you would send useful information to standard out, such as the text encoding?

I decided to not do this for now but made a note about the possibility in #12

See also https://github.com/neilpa/yajsv/issues/13#issuecomment-694932174 for adding additional output such as the text encoding in a verbose mode.

Adds support to detect and decode UTF-16 (BE or LE) JSON text. Also adds an option to skip BOMs at the start of a JSON document. These are treated as errors by default (with a better message) based on RFC 8259. Resolves #13

TODO

• [x] Tests for the data generated in 6f93deb for the new encoding combos
• [ ] Tests for one and two byte JSON docs

I've been running some tests using your yajsv (windows 64 version) utility, and I've found that both UTF-8 (BOM) and UTF-16 LE encoded files throw the following errors:

UTF-8 with BOM : error: validate: invalid character 'ï' looking for beginning of value UTF-16 LE : error: validate: invalid character 'ÿ' looking for beginning of value

For my purpose implementing in a PowerShell script, it's not a big deal since I can re-encode the files to standard UTF-8 and it works, but I thought you'd like to know about the encoding limitations.

Great utility, BTW!

Exporting the rendered edit via PhotoKit would be ideal. I'm pretty sure I saw an option for that as part of fetching asset image data but now I'm struggling to find it again in the PhotoKit docs.

If that doesn't work, it should be possible to apply all the adjustments now that I've figured out the schema and how to extract the associated dictionaries. There's a lot of them though so it's going to be non-trivial to map them all onto onto CoreImage operations.

I checked and the edited JPEGs exist on Catalina under resources/renders/${UUID_PREFIX}/${UUID}_1_201_a.jpeg. (Didn't check the DB though to see if that's referenced somewhere).

Could re-open this now based on the progress in https://github.com/RhetTbull/osxphotos/issues/215#issuecomment-698663130. Would also keep that issue focused on RAW conversion and could further discuss how to handle adjustment data here.

I did more digging into the adjustment data and seem to have figured out the schema for the full scope of edits that are possible in the Photos app. There's a lot of details in https://github.com/neilpa/photohack/pull/1. I haven't done much testing yet on sample images but would be interesting to run this through the test libraries in this repo.

I also figured out how to make a standalone binary work so I attached that to a release. That should make it a bit easier to experiment with. For now I just have it dump a JSON map with the asset UUID as keys and the adjustment dictionaries as values.

Or, the interface could be a local webserver with RESTful API though that would take more work than a simple CLI.

That would be an interesting approach but I'm not sure how simple it is to do something like this in Swift/Objective-C out of the box. I usually default to Go for CLI apps for this reason, it's trivial to add a web/http interface if needed. In this case I didn't want to add yet another language into the mix.

The README contains most of the details. This decodes the adjustment data as a JSON map for a given set of asset UUIDs.

Additionally there's notes, links, debug spew, and helper function that extracted the entire array of ajdustment possibilities in the Photos app. Note the "array" and "compound" adjustments may not be right as those are more complicated and haven't been tested yet.

The README contains most of the details. This decodes the adjustment data as a JSON map for a given set of asset UUIDs.

Additionally there's notes, links, debug spew, and helper function that extracted the entire array of ajdustment possibilities in teh Photos app. Note the "array" and "compound" adjustments may not be right as those are more complicated and haven't been tested yet.

Here's my initial Poc Xcode project - https://github.com/neilpa/photohack

If you add a new Xcode scheme with the first argument as a path to a *.photoslibrary and subsequent arguments as asset UUIDs it'll dump adjustment data in a readable format. That should get you something like the following.

{
    Album = PHAssetCollection;
    Asset = PHAsset;
    CloudSharedAlbum = PHCloudSharedAlbum;
    DetectedFace = PHFace;
    DetectedFaceGroup = PHFaceGroup;
    FaceCrop = PHFaceCrop;
    FetchingAlbum = PHAssetCollection;
    Folder = PHCollectionList;
    GenericAsset = PHAsset;
    ImportSession = PHImportSession;
    Keyword = PHKeyword;
    LegacyFaceAlbum = PHAssetCollection;
    Memory = PHMemory;
    Moment = PHMoment;
    MomentList = PHMomentList;
    MomentShare = PHMomentShare;
    MomentShareParticipant = PHMomentShareParticipant;
    Person = PHPerson;
    PhotoStreamAlbum = PHAssetCollection;
    PhotosHighlight = PHPhotosHighlight;
    ProjectAlbum = PHProject;
    Question = PHQuestion;
    Suggestion = PHSuggestion;
}
adjustments: E1C9A934-260F-4211-95AD-14827A82ACB4
  <none>
2020-09-22 20:17:59.046444-0700 photohack[13032:3222620] Metal API Validation Enabled
adjustments: 34B8483F-8A35-4FEB-98A3-84DC376F0C9F
  composition = <NUGenericComposition:0x100457b60 id=me.neilpa.photohack:PhotosComposition~1.0 mediaType=Unknown contents={
	raw = <NUGenericAdjustment:0x100458080> id=me.neilpa.photohack:RAW~1.0 settings={
    auto = 0;
    enabled = 1;
    inputDecoderVersion = 8;
},
	cropStraighten = <NUGenericAdjustment:0x1004582e0> id=me.neilpa.photohack:CropStraighten~1.0 settings={
    angle = "-0";
    auto = 0;
    constraintHeight = 0;
    constraintWidth = 0;
    enabled = 1;
    height = 988;
    width = 1482;
    xOrigin = 2342;
    yOrigin = 1127;
},
	orientation = <NUGenericAdjustment:0x100458610> id=me.neilpa.photohack:Orientation~1.0 settings={
    value = 1;
},
}>

I'll try to get a proper binary built (and updated instructions) but haven't figured out how to properly embed the Info.plist such that it can request photos library permissions.

There's also https://raw.pixls.us/ which is the successor to https://rawsamples.ch

Yea, you definitely want to reuse the context if possible. (There was a similar comment the the SO post I linked).

I think PhotoKit will only work with the "system library" and many people have more than one Photos library.

After spending some time with the private headers I was able to hack together some Objective-C code that successfully loads a photos library from an arbitrary path. I got far enough to fetch a PHAsset via UUID and validate a few properties, including the path on disk is expected.

I need to clean this up a bit but I'll push a proof-of-concept Xcode project and share more of my findings. There's some interesting stuff in the header dump that I want to look at more. In particular, I think I've found a way to extract the adjustment data (at least the crop region along with orientation and rotation).

Good call on CoreImage, that makes the conversion nearly trivial. I did a quick test with that approach and it worked great for the raw images I tried.

As for the PhotoKit limitation, it may be possible to use a custom library via the private header initialization methods. I'm one of those folks with a non-standard library and will see if I can get something working here.

I hadn't actually used dcraw before and was doing some experiments this morning. I'm not sure it's going to work as well as I had hoped, at least for bulk exports. The default color correction leaves a lot to be desired. I also experimented with various options and got slightly better results. However, it doesn't compare to the default results I get from the Photos app (or something like Darktable). I'm very much a noob at "developing" digital photos so I may be overlooking something obvious.

There's also darktable-cli which I want to test. Maybe that will work better out of the box.

I'm not competent in Swift or Objective-C

It's been awhile, but I've used both a fair bit. I may experiment this afternoon to see what's possible in terms of export.

This is the answer I was afraid of.

For .heic, /usr/bin/sips can do the conversion

Interesting, I wasn't aware of this built-in command and had been using the libheif tools

Photos itself can convert the images.

Yea, I had a small hope that this would be available via PhotoKit without having to script the GUI. I can't imagine going that route once you start getting above 1k photos. Exporting of my full library of ~16k photos to a fast external SSD and that takes ~20 minutes currently. Seems unlikely that would go well when invoking Photos for each image given the occassional crashes I already see.

Another possibility would be to add an option to allow the user to run a command of choice

I suspect this would be hard to make work in practice. You'll likely need different commands for heif (e.g. sips) vs raw (e.g. dcraw) files. You could wrap it in a shell script, but then you lose the benefits of the templating system. Maybe there's an ergonomic way to do this on an per-extension (or UTI?) basis. However, I worry the plethora of raw file formats would make this impractical.

the resulting converted photo would not be tracked by osxphotos so it would not be considered when running export --updated

I haven't looked into the mechanics of how --updated works yet. My current thinking (sans builtin conversion) is to do an export, do a post-hoc conversion, then truncate the RAW files to 0 bytes keeping the exported file created/mod times. Would that "trick" osxphotos into not re-exporting images unless they had actually been updated in the Photos app?

Would it be feasible to add an option to always convert RAW images on export to JPEG? Any edited photos are already going to be JPEG but I've got a lot of un-edited raws w/out the JPEG preview. This would allow something like the following (totally made up --raw-to-jpeg) to quickly create a backup that is also extremely portable.

osxphotos export --raw-to-jpeg --export-by-date \
  --skip-original-if-edited --edit-suffix "" \
  --no-extended-attributes /Volumes/thumbdrive

Adds support to detect and decode UTF-16 (BE or LE) JSON text. Also adds an option to skip BOMs at the start of a JSON document. These are treated as errors by default (with a better message) based on RFC 8259. Resolves #13

TODO

• [ ] Tests for the data generated in 6f93deb for the new encoding combos
• [ ] Tests for one and two byte JSON docs

Thanks for looking more closely at the RFCs. Obviously I didn't read those carefully enough last night. Given that second sentence it seems reasonable to allow it.

you could also consider it a warning condition and maybe add a switch to ignore warnings, leaving the default to treat them as errors.

Right now files are bucketed to a pass, fail, or error, the last being invalid JSON. I'd say these should still default to an "error" but with a better message about the BOM. The rationale being the moment they are served as-is over the internet the become invalid.

As for the switch, I'm thinking along the lines of -allow-bom to opt-out of that behavior. Although that would diverge from my current single character flags for everything. Might be time to rethink/break given the feedback in #12

Further complicating matters, YAML does support BOMs and required for UTF-16 detection.

Turns out it's invalid to include a BOM in a JSON document. I'll make this case fail with a better error message but it doesn't make sense for a JSON validator to accept invalid JSON :wink:. As for the other UTF-* encodings, that post offers a simple approach to detect them based on the first few bytes.

In Catalina, Finder does show the "/" in place of the ":"

Woah, crazy that Finder blindly replaces : in a directory name with / for display. Guess this originates from the pre-OSX days which was before I switched to macs.

I had forgotten about the prevalence of UTF-16 on Windows although I can't say I've ever seen UTF-8 files with a BOM. I suspect this would be relatively easy to detect and support via the golang.org/x/text package.

Out of curiosity, do you happen to know what utilities generated the UTF-8 w/ BOM files?

Both seems like the right approach to me. I know that darktable defaults to appending .xmp while keeping the original file extension.

I've not been able to find a way to test this. I've tried both Digikam and Lightroom, both of which I'm unfamiliar with, and can't figure out how to get them to import face regions from an XMP file despite hours of googling

I hit this exact same issue when working on this in phace. I could not find an app that actually used the encoded face regions. IIRC the docs for DigiKam claimed to but in all my experimenting I couldn't find an example of it working. Best I did was find some sample images with the XMP face regions already encoded and made sure I could round-trip the data based on my understanding of the format.