Add iOS Live Photo support and another format for guess_extractor #224

BobTheSoftwareDeveloper · 2023-07-16T12:00:10Z

First of all, great tool! It has made my process of organising the Google Photos export so much nicer for when I upload the backup to S3.

I have tested the changes locally and it was able to add the EXIF date data to the video files successfully.

BobTheSoftwareDeveloper · 2023-07-16T12:01:02Z

lib/date_extractors/guess_extractor.dart

@@ -43,6 +43,12 @@ final _commonDatetimePatterns = [
        r'(?<date>(20|19|18)\d{2}_(01|02|03|04|05|06|07|08|09|10|11|12)_[0-3]\d_\d{2}_\d{2}_\d{2})'),
    'YYYY_MM_DD_hh_mm_ss',
  ],
+  // example: 20190620_184109.jpg


I found this format of photos when going through my old photos 20190620_184109.jpg, it seems to be used by older cameras.

BobTheSoftwareDeveloper · 2023-07-16T12:04:09Z

lib/date_extractors/heic_video_extractor.dart

+  final fileNameWithoutExtension = p.basenameWithoutExtension(file.path);
+
+  if (fileNameWithoutExtension.startsWith("IMG") &&
+      p.extension(file.path).toLowerCase() == ".mp4") {


Photo from iPhone with Live Photo enabled when uploading to Google Photos will create three files:

IMG_1845.HEIC

IMG_1845.MP4

IMG_1845.HEIC.json

The .MP4 file always comes in a pair with the .HEIC file when the live photo is enabled, however, there is no JSON file for the video. We can also use the JSON file for the image to set the date for the video. This will ensure that the image and the video are stored in the same directory.

I noticed that it doesn't always start with IMG, for example, one of mine is like this:

97E2514D-F5BB-4314-B099-9A5FAE5EEA22.JPG 97E2514D-F5BB-4314-B099-9A5FAE5EEA22.JPG.json 97E2514D-F5BB-4314-B099-9A5FAE5EEA22.MP4

I too see other files. Perhaps instead try to probe whether the img file exists next to it by doing an exists check on the "possibleFileExtensions".

BobTheSoftwareDeveloper · 2023-07-16T12:08:12Z

lib/date_extractors/heic_video_extractor.dart

+    "jpg",
+    "JPEG",
+    "jpeg"
+  ];


I have come across a few situations when the original image in the live photo pair is saved as a .jpg or .jpeg, they seem to use both upper and lower case. I suspect this is either because the HEIC format is disabled or the photo with the live photo feature was sent over iMessage which converted the image into a .jpeg format.

Sample directory of the files:

IMG_3984.jpeg

IMG_3984.MP4

IMG_3984.jpeg.json

and...

IMG_7334.JPG

IMG_7334.MP4

IMG_7334.JPG.json

BobTheSoftwareDeveloper

Done adding comments, requesting for your review :) @TheLastGimbus

TheLastGimbus · 2023-08-13T22:59:31Z

Nice! Tho i will need to think more about it, how we should handle it etc, so give me some time to merge it...

TheLastGimbus · 2023-09-23T00:57:27Z

So, for now, i'm leaving this waiting for further advancements in metioned discussion...

But, for anyway that wants this NOW, here are nightlies :>

https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper/actions/runs/6280579316

dbrennand · 2023-09-25T14:45:19Z

Hey @TheLastGimbus @BobTheSoftwareDeveloper

Thanks for putting this PR together, I just tested the nightly build and wow! What a difference!

Using the latest version (v3.4.3) gpth shows loads of errors like:

Can't get date on /Volumes/Macintosh HD/Users/user/Downloads/Takeout/Google Photos/.../IMG_8354.MP4

And the summary output shows gpth failing to find metadata for many files:

Couldn't find date for 1997 photos/videos :/

Whereas, using the nightly build with this PR included:

Finding duplicates...
Guessing dates from files : █████████████████....................... 5285/12187  
Can't get date on /Volumes/Macintosh HD/Users/user/Downloads/Takeout/Google Photos/Photos from 2019/moments_09_19_8_20_51 pm.MP4
Guessing dates from files : ███████████████████████████████████████. 12186/12187  
Finding albums (this may take some time, dont worry :) ...
Moving photos to output folder : ████████████████████████████████████████ 12187/12187  
========================================
DONE! FREEEEEDOOOOM!!!
Skipped 3 duplicates
Couldn't find date for 1 photos/videos :/

Only 1 file failed 👍🏻 😃

TheLastGimbus · 2023-09-25T15:21:37Z

Only 1 file failed

We're leaving not even one! @dbrennand Could you try to find other files related to this one (og photo and json)?

dbrennand · 2023-09-25T15:26:47Z

Only 1 file failed

We're leaving not even one! @dbrennand Could you try to find other files related to this one (og photo and json)?

I can have a look, what am I looking for exactly in the JSON?

TheLastGimbus · 2023-09-25T21:12:18Z

You're just looking for the json, and it's filename 👍👍

I just want to figure out why gpth didn't find it - it does so by filenames purely

dbrennand · 2023-09-26T21:30:08Z

You're just looking for the json, and it's filename 👍👍

I just want to figure out why gpth didn't find it - it does so by filenames purely

@TheLastGimbus - I'm not sure why it failed... 🤔 So for this live photo, the three files are named:

moments_09_19_8_20_51 pm.JPG
moments_09_19_8_20_51 pm.MP4
moments_09_19_8_20_51 pm.JPG.json

title value in the JSON is:

{
  "title": "moments_09_19_8_20_51 pm.JPG",
}

zxsleebu · 2023-12-06T19:33:26Z

can you please look into it?

TheLastGimbus · 2023-12-07T08:44:54Z

@zxsleebu I'm gonna be honest - right now, no.

And I can't really point when, but like a month from now at least. If you need your takeout with this, use @BobTheSoftwareDeveloper fork - nightlies are still here: https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper/actions/runs/6280579316

It will even help to test it if you used it. I just don't have the time right now to make 10000% sure it all works (and potentially add all cool features above) and list it on the official repo

Alkanov · 2023-12-25T20:11:09Z

can you please re-build https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper/actions/runs/6280579316

Edit: nvm I forked it and ran it myself. I can confirm I went from 3.5k "Can't get date on...." to only 40 - everything else worked fine. Thanks

TheLastGimbus · 2023-12-29T16:30:58Z

https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper/actions/runs/7358237528

pzh200707 · 2024-04-01T08:59:09Z

https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper/actions/runs/7358237528

I have met the same problem. Latest release is on Sep 23, 2023. The rebuild has expired. Can we have a new release?

TheLastGimbus · 2024-04-01T15:45:07Z

Sure guys

https://github.com/TheLastGimbus/GooglePhotosTakeoutHelper/actions/runs/8510134630

janhoy · 2024-04-17T10:56:26Z

bin/gpth.dart

@@ -125,6 +126,7 @@ void main(List<String> arguments) async {
    jsonExtractor,
    exifExtractor,
    if (args['guess-from-name']) guessExtractor,
+    heicVideoExtractor,


Should this extractor perhaps come before the guessExtractor? If it exits quickly when conditions are not met, it should be safe.

mguzman011 · 2024-05-19T21:42:46Z

Any chance of being able to add RAW file support? Seems like some .dng and .cr2 files I had weren't transferred over.

Not the worst, since it seems like the EXIF data has been kept intact, but adds another step to the process.

Edit: Seems like .cr2 files were transferred over fine. What caused me issues were ~3mb "CR2" files that were actually jpgs, but labeled CR2 thanks to Google Photos. Renaming the files as JPG brought back all of their EXIF data and I could organize accordingly.

TheLastGimbus · 2024-05-20T11:11:16Z

Seems like some .dng and .cr2 files I had weren't transferred over.

I will look at it when i will take care of this project some day 🌈

BobTheSoftwareDeveloper added 2 commits July 16, 2023 23:51

Add iOS Live Photo support

3944b11

Add another format for photo guess_extractor

33a3756

BobTheSoftwareDeveloper commented Jul 16, 2023

View reviewed changes

BobTheSoftwareDeveloper commented Jul 18, 2023

View reviewed changes

BobTheSoftwareDeveloper mentioned this pull request Jul 21, 2023

Live Photos / HEICs #225

Closed

TheLastGimbus mentioned this pull request Sep 11, 2023

Handle "moving photo" video files #180

Open

janhoy reviewed Apr 17, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add iOS Live Photo support and another format for guess_extractor #224

Add iOS Live Photo support and another format for guess_extractor #224

BobTheSoftwareDeveloper commented Jul 16, 2023 •

edited

BobTheSoftwareDeveloper Jul 16, 2023

BobTheSoftwareDeveloper Jul 16, 2023

andrewchen5678 Sep 4, 2023

janhoy Apr 17, 2024

BobTheSoftwareDeveloper Jul 16, 2023

BobTheSoftwareDeveloper left a comment

TheLastGimbus commented Aug 13, 2023

TheLastGimbus commented Sep 23, 2023

dbrennand commented Sep 25, 2023

TheLastGimbus commented Sep 25, 2023

dbrennand commented Sep 25, 2023

TheLastGimbus commented Sep 25, 2023

dbrennand commented Sep 26, 2023

zxsleebu commented Dec 6, 2023

TheLastGimbus commented Dec 7, 2023 •

edited

Alkanov commented Dec 25, 2023 •

edited

TheLastGimbus commented Dec 29, 2023

pzh200707 commented Apr 1, 2024

TheLastGimbus commented Apr 1, 2024

janhoy Apr 17, 2024

mguzman011 commented May 19, 2024 •

edited

TheLastGimbus commented May 20, 2024

Add iOS Live Photo support and another format for guess_extractor #224

Are you sure you want to change the base?

Add iOS Live Photo support and another format for guess_extractor #224

Conversation

BobTheSoftwareDeveloper commented Jul 16, 2023 • edited

BobTheSoftwareDeveloper Jul 16, 2023

Choose a reason for hiding this comment

BobTheSoftwareDeveloper Jul 16, 2023

Choose a reason for hiding this comment

andrewchen5678 Sep 4, 2023

Choose a reason for hiding this comment

janhoy Apr 17, 2024

Choose a reason for hiding this comment

BobTheSoftwareDeveloper Jul 16, 2023

Choose a reason for hiding this comment

BobTheSoftwareDeveloper left a comment

Choose a reason for hiding this comment

TheLastGimbus commented Aug 13, 2023

TheLastGimbus commented Sep 23, 2023

dbrennand commented Sep 25, 2023

TheLastGimbus commented Sep 25, 2023

dbrennand commented Sep 25, 2023

TheLastGimbus commented Sep 25, 2023

dbrennand commented Sep 26, 2023

zxsleebu commented Dec 6, 2023

TheLastGimbus commented Dec 7, 2023 • edited

Alkanov commented Dec 25, 2023 • edited

TheLastGimbus commented Dec 29, 2023

pzh200707 commented Apr 1, 2024

TheLastGimbus commented Apr 1, 2024

janhoy Apr 17, 2024

Choose a reason for hiding this comment

mguzman011 commented May 19, 2024 • edited

TheLastGimbus commented May 20, 2024

BobTheSoftwareDeveloper commented Jul 16, 2023 •

edited

TheLastGimbus commented Dec 7, 2023 •

edited

Alkanov commented Dec 25, 2023 •

edited

mguzman011 commented May 19, 2024 •

edited