Google media exporter: installs along side Photos, but will replace photos in the future (#1218)

`GoogleMediaExporter` class mosty written by seehamrun@ with later additional commits by jzacsh@

* Create MediaContainerResource to be used by the new MediaExporters and MediaImporters

      This combines the photo and video resources

* Account for videos in the transmogrification logic

* Fix typo in the Transmogrification config

* Add Video limits to transmogrification config

* add clean title to the transmogrification config for videos

* undo rename of the videomodel (let this get handled when the PR to rename gets pushed)

* fix warning abut overriding equals but not hashcode

* remove logic to split up the albums in the media resource

* slight fix to the transmogrification function -- move up the root album to the main function

* Update MediaContainer to use the new VideoModel instead of VideoObject

* Use Album and Video transmogrification specific variables

* Fix return case when maxLength <= 0

* Initial skeleton of the GoogleMediaExporter

This is currently just a copy of the GooglePhotosExporter without all of the content for exporting videos

* Hookup the mediaexporter to the transfer extension

* Add support for video exports in the GoogleMediaExporter

* fix importer

* move convertToVideoModel and convertToPhotoModel into the GoogleMediaItem class

* use getFileName as the title for videos

* Rename PopulateContainedPhotosList method to PopulateContainedMediaList

* Rename `exportPhotos` to `exportMedia`

* Add video support for the exportMediaContainerResource function

jzacsh@'s continued commits:

* fix broken tests: have new data & exceptions

both video and photo models have a new "upload time" field so this
commit _starts_ to introduce that.

also tweaked some of the existing code that was written before
Upload...Exception was added to some signatures.

* nit: consistency in inline-documentation

* nit: stick to existing pattern (of always having a setter)

* noop(docs) leave trail to cleanup since this PR is forking code

* noop(refactor) mv Temp{Photo,Media}Data

noop as this is just a rename of current temp-store logic that is
hardcoded to Photos. Later work will come along to extend the
TempMediaData object to handle new data (added TODO to this point atop
the class).

* WIP: fork of Photos tests trivial "passing"


* noop(refactor) make name match the forked purpose of this token prefix

* WIP: sets roadmap to complete unit test

comparing to the history of the forked Exporter _itself_ - the core
thing that changed was adding video, but most of the code stayed
identical. this unit test then leaves the core "photo set" unit test in
place and simply adds another for video, trying to mirror the fork of
the exporter itself as closely as possible.

* noop(prefactor) clarify photo-logic in tests

this clarifies what the photo-specific fakes are between a unit test's
arrange and assert phases. that's not always clear for _other_ consts
that are pulled to the top of the test file, but I'm trying to merely
unblock the addition of video logic here (not do a _general_ cleanup).

* bugfix: all the fakes should be readonly

* finishes unit test by fixing URL assertion

while I'm at it, also refactoring the Media objects involved so google
adapter logic isn't doing deep property-null-checking as an API (`isVideo`
`isPhoto`).

---------

Co-authored-by: Siham Hussein <sihamh@google.com>

Author: jzacsh

extensions/data-transfer/portability-data-transfer-google/src/main/java/org/datatransferproject/datatransfer/google/GoogleTransferExtension.java

@@ -4,6 +4,7 @@
 import static org.datatransferproject.types.common.models.DataVertical.CALENDAR;
 import static org.datatransferproject.types.common.models.DataVertical.CONTACTS;
 import static org.datatransferproject.types.common.models.DataVertical.MAIL;
+import static org.datatransferproject.types.common.models.DataVertical.MEDIA;
 import static org.datatransferproject.types.common.models.DataVertical.PHOTOS;
 import static org.datatransferproject.types.common.models.DataVertical.SOCIAL_POSTS;
 import static org.datatransferproject.types.common.models.DataVertical.TASKS;
@@ -27,6 +28,7 @@
 import org.datatransferproject.datatransfer.google.gplus.GooglePlusExporter;
 import org.datatransferproject.datatransfer.google.mail.GoogleMailExporter;
 import org.datatransferproject.datatransfer.google.mail.GoogleMailImporter;
+import org.datatransferproject.datatransfer.google.media.GoogleMediaExporter;
 import org.datatransferproject.datatransfer.google.photos.GooglePhotosExporter;
 import org.datatransferproject.datatransfer.google.photos.GooglePhotosImporter;
 import org.datatransferproject.datatransfer.google.tasks.GoogleTasksExporter;
@@ -50,7 +52,7 @@ public class GoogleTransferExtension implements TransferExtension {
   // TODO: centralized place, or enum type for these
   private static final ImmutableList<DataVertical> SUPPORTED_SERVICES =
       ImmutableList.of(
-          BLOBS, CALENDAR, CONTACTS, MAIL, PHOTOS, SOCIAL_POSTS, TASKS, VIDEOS);
+          BLOBS, CALENDAR, CONTACTS, MAIL, PHOTOS, SOCIAL_POSTS, TASKS, VIDEOS, MEDIA);
   private ImmutableMap<DataVertical, Importer> importerMap;
   private ImmutableMap<DataVertical, Exporter> exporterMap;
   private boolean initialized = false;
@@ -132,6 +134,8 @@ public void initialize(ExtensionContext context) {
     exporterBuilder.put(
         PHOTOS, new GooglePhotosExporter(credentialFactory, jobStore, jsonFactory, monitor));
     exporterBuilder.put(VIDEOS, new GoogleVideosExporter(credentialFactory, jsonFactory));
+    exporterBuilder.put(
+        MEDIA, new GoogleMediaExporter(credentialFactory, jobStore, jsonFactory, monitor));
 
     exporterMap = exporterBuilder.build();
 

extensions/data-transfer/portability-data-transfer-google/src/main/java/org/datatransferproject/datatransfer/google/media/GoogleMediaExporter.java

@@ -17,10 +17,13 @@
 package org.datatransferproject.datatransfer.google.mediaModels;
 
 import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.base.Preconditions;
+import java.util.Date;
+import java.util.Optional;
+import org.datatransferproject.types.common.models.photos.PhotoModel;
+import org.datatransferproject.types.common.models.videos.VideoModel;
 
-/**
- Media item returned by queries to the Google Photos API.  Represents what is stored by Google.
- */
+/** Media item returned by queries to the Google Photos API. Represents what is stored by Google. */
 public class GoogleMediaItem {
   @JsonProperty("id")
   private String id;
@@ -43,31 +46,118 @@ public class GoogleMediaItem {
   @JsonProperty("productUrl")
   private String productUrl;
 
-  public String getId() { return id; }
+  @JsonProperty("uploadedTime")
+  private Date uploadedTime;
 
-  public String getDescription() { return description; }
+  public boolean isPhoto() {
+    return this.getMediaMetadata().getPhoto() != null;
+  }
+  public boolean isVideo() {
+    return this.getMediaMetadata().getVideo() != null;
+  }
 
-  public String getBaseUrl() { return baseUrl; }
 
-  public String getMimeType() { return mimeType; }
+  public String getFetchableUrl() {
+    if (this.isPhoto()) {
+      return this.getBaseUrl() + "=d";
+    } else if (this.isVideo()) {
+      // dv = download video otherwise you only get a thumbnail
+      return this.getBaseUrl() + "=dv";
+    } else {
+      throw new IllegalArgumentException("unimplemented media type");
+    }
+  }
 
-  public String getFilename() { return filename; }
+  public static VideoModel convertToVideoModel(
+      Optional<String> albumId, GoogleMediaItem mediaItem) {
+    Preconditions.checkArgument(mediaItem.isVideo());
+
+    return new VideoModel(
+        mediaItem.getFilename(),
+        mediaItem.getFetchableUrl(),
+        mediaItem.getDescription(),
+        mediaItem.getMimeType(),
+        mediaItem.getId(),
+        albumId.orElse(null),
+        false /*inTempStore*/,
+        mediaItem.getUploadedTime());
+  }
 
-  public String getProductUrl() {
-    return productUrl;
+  public static PhotoModel convertToPhotoModel(
+      Optional<String> albumId, GoogleMediaItem mediaItem) {
+    Preconditions.checkArgument(mediaItem.isPhoto());
+
+    return new PhotoModel(
+        mediaItem.getFilename(),
+        mediaItem.getFetchableUrl(),
+        mediaItem.getDescription(),
+        mediaItem.getMimeType(),
+        mediaItem.getId(),
+        albumId.orElse(null),
+        false  /*inTempStore*/,
+        null  /*sha1*/,
+        mediaItem.getUploadedTime());
+  }
+
+  public String getId() {
+    return id;
+  }
+
+  public void setId(String id) {
+    this.id = id;
+  }
+
+  public String getDescription() {
+    return description;
+  }
+
+  public void setDescription(String description) {
+    this.description = description;
+  }
+
+  public String getBaseUrl() {
+    return baseUrl;
+  }
+
+  public void setBaseUrl(String baseUrl) {
+    this.baseUrl = baseUrl;
+  }
+
+  public String getMimeType() {
+    return mimeType;
+  }
+
+  public void setMimeType(String mimeType) {
+    this.mimeType = mimeType;
   }
 
-  public MediaMetadata getMediaMetadata() { return mediaMetadata; }
+  public String getFilename() {
+    return filename;
+  }
 
-  public void setDescription(String description) { this.description = description; }
+  public void setFilename(String filename) {
+    this.filename = filename;
+  }
 
-  public void setBaseUrl(String baseUrl) { this.baseUrl = baseUrl; }
+  // TODO(zacsh) investigate why/if there's no setter for this; do we need setters or does the java
+  // annotation do the work for us somehow?
+  public String getProductUrl() {
+    return productUrl;
+  }
 
-  public void setId(String id) { this.id = id; }
+  public MediaMetadata getMediaMetadata() {
+    return mediaMetadata;
+  }
 
-  public void setMimeType(String mimeType) { this.mimeType = mimeType; }
+  public void setMediaMetadata(MediaMetadata mediaMetadata) {
+    this.mediaMetadata = mediaMetadata;
+  }
 
-  public void setFilename(String filename) { this.filename = filename; }
+  public Date getUploadedTime() {
+    return this.uploadedTime;
+  }
 
-  public void setMediaMetadata(MediaMetadata mediaMetadata) { this.mediaMetadata = mediaMetadata; }
+  public void setUploadedTime(Date date) {
+    this.uploadedTime = date;
+  }
 }

extensions/data-transfer/portability-data-transfer-google/src/main/java/org/datatransferproject/datatransfer/google/mediaModels/GoogleMediaItem.java

@@ -44,7 +44,7 @@
 import org.datatransferproject.spi.transfer.types.ContinuationData;
 import org.datatransferproject.spi.transfer.types.InvalidTokenException;
 import org.datatransferproject.spi.transfer.types.PermissionDeniedException;
-import org.datatransferproject.spi.transfer.types.TempPhotosData;
+import org.datatransferproject.spi.transfer.types.TempMediaData;
 import org.datatransferproject.spi.transfer.types.UploadErrorException;
 import org.datatransferproject.types.common.ExportInformation;
 import org.datatransferproject.types.common.PaginationData;
@@ -58,6 +58,8 @@
 // Not ready for prime-time!
 // TODO: fix duplication problems introduced by exporting all photos in 'root' directory first
 
+// TODO WARNING DO NOT MODIFY THIS CLASS! (unless you're willing to mirror your changes to
+// GoogleMediaExporter too). This class is deprecated in favor. TODO here is to delete this class.
 public class GooglePhotosExporter
     implements Exporter<TokensAndUrlAuthData, PhotosContainerResource> {
 
@@ -165,7 +167,7 @@ private ExportResult<PhotosContainerResource> exportPhotosContainer(
     for (PhotoModel photo : container.getPhotos()) {
       GoogleMediaItem googleMediaItem =
           getOrCreatePhotosInterface(authData).getMediaItem(photo.getDataId());
-      photosBuilder.add(convertToPhotoModel(Optional.empty(), googleMediaItem));
+      photosBuilder.add(GoogleMediaItem.convertToPhotoModel(Optional.empty(), googleMediaItem));
     }
 
     PhotosContainerResource photosContainerResource =
@@ -274,8 +276,8 @@ ExportResult<PhotosContainerResource> exportPhotos(
   void populateContainedPhotosList(UUID jobId, TokensAndUrlAuthData authData)
       throws IOException, InvalidTokenException, PermissionDeniedException, UploadErrorException {
     // This method is only called once at the beginning of the transfer, so we can start by
-    // initializing a new TempPhotosData to be store in the job store.
-    TempPhotosData tempPhotosData = new TempPhotosData(jobId);
+    // initializing a new TempMediaData to be store in the job store.
+    TempMediaData tempMediaData = new TempMediaData(jobId);
 
     String albumToken = null;
     AlbumListResponse albumListResponse;
@@ -293,7 +295,7 @@ void populateContainedPhotosList(UUID jobId, TokensAndUrlAuthData authData)
                     .listMediaItems(Optional.of(albumId), Optional.ofNullable(photoToken));
             if (containedMediaSearchResponse.getMediaItems() != null) {
               for (GoogleMediaItem mediaItem : containedMediaSearchResponse.getMediaItems()) {
-                tempPhotosData.addContainedPhotoId(mediaItem.getId());
+                tempMediaData.addContainedPhotoId(mediaItem.getId());
               }
             }
             photoToken = containedMediaSearchResponse.getNextPageToken();
@@ -305,7 +307,7 @@ void populateContainedPhotosList(UUID jobId, TokensAndUrlAuthData authData)
 
     // TODO: if we see complaints about objects being too large for JobStore in other places, we
     // should consider putting logic in JobStore itself to handle it
-    InputStream stream = convertJsonToInputStream(tempPhotosData);
+    InputStream stream = convertJsonToInputStream(tempMediaData);
     jobStore.create(jobId, createCacheKey(), stream);
   }
 
@@ -332,10 +334,10 @@ private List<PhotoModel> convertPhotosList(
       Optional<String> albumId, GoogleMediaItem[] mediaItems, UUID jobId) throws IOException {
     List<PhotoModel> photos = new ArrayList<>(mediaItems.length);
 
-    TempPhotosData tempPhotosData = null;
+    TempMediaData tempMediaData = null;
     InputStream stream = jobStore.getStream(jobId, createCacheKey()).getStream();
     if (stream != null) {
-      tempPhotosData = new ObjectMapper().readValue(stream, TempPhotosData.class);
+      tempMediaData = new ObjectMapper().readValue(stream, TempMediaData.class);
       stream.close();
     }
 
@@ -344,12 +346,12 @@ private List<PhotoModel> convertPhotosList(
         // TODO: address videos
         boolean shouldUpload = albumId.isPresent();
 
-        if (tempPhotosData != null) {
-          shouldUpload = shouldUpload || !tempPhotosData.isContainedPhotoId(mediaItem.getId());
+        if (tempMediaData != null) {
+          shouldUpload = shouldUpload || !tempMediaData.isContainedPhotoId(mediaItem.getId());
         }
 
         if (shouldUpload) {
-          PhotoModel photoModel = convertToPhotoModel(albumId, mediaItem);
+          PhotoModel photoModel = GoogleMediaItem.convertToPhotoModel(albumId, mediaItem);
           photos.add(photoModel);
 
           monitor.debug(
@@ -360,19 +362,6 @@ private List<PhotoModel> convertPhotosList(
     return photos;
   }
 
-  private PhotoModel convertToPhotoModel(Optional<String> albumId, GoogleMediaItem mediaItem) {
-    Preconditions.checkArgument(mediaItem.getMediaMetadata().getPhoto() != null);
-
-    return new PhotoModel(
-        mediaItem.getFilename(),
-        mediaItem.getBaseUrl() + "=d",
-        mediaItem.getDescription(),
-        mediaItem.getMimeType(),
-        mediaItem.getId(),
-        albumId.orElse(null),
-        false);
-  }
-
   private synchronized GooglePhotosInterface getOrCreatePhotosInterface(
       TokensAndUrlAuthData authData) {
     return photosInterface == null ? makePhotosInterface(authData) : photosInterface;
@@ -385,6 +374,6 @@ private synchronized GooglePhotosInterface makePhotosInterface(TokensAndUrlAuthD
   }
 
   private static String createCacheKey() {
-    return "tempPhotosData";
+    return "tempMediaData";
   }
 }

extensions/data-transfer/portability-data-transfer-google/src/main/java/org/datatransferproject/datatransfer/google/photos/GooglePhotosExporter.java

@@ -95,7 +95,7 @@ public class GooglePhotosInterface {
   private final GoogleCredentialFactory credentialFactory;
   private final RateLimiter writeRateLimiter;
 
-  GooglePhotosInterface(
+  public GooglePhotosInterface(
       GoogleCredentialFactory credentialFactory,
       Credential credential,
       JsonFactory jsonFactory,
@@ -108,7 +108,7 @@ public class GooglePhotosInterface {
     writeRateLimiter = RateLimiter.create(writesPerSecond);
   }
 
-  AlbumListResponse listAlbums(Optional<String> pageToken)
+  public AlbumListResponse listAlbums(Optional<String> pageToken)
       throws IOException, InvalidTokenException, PermissionDeniedException {
     Map<String, String> params = new LinkedHashMap<>();
     params.put(PAGE_SIZE_KEY, String.valueOf(ALBUM_PAGE_SIZE));
@@ -118,18 +118,18 @@ AlbumListResponse listAlbums(Optional<String> pageToken)
     return makeGetRequest(BASE_URL + "albums", Optional.of(params), AlbumListResponse.class);
   }
 
-  GoogleAlbum getAlbum(String albumId) throws IOException, InvalidTokenException, PermissionDeniedException{
+  public GoogleAlbum getAlbum(String albumId) throws IOException, InvalidTokenException, PermissionDeniedException{
     Map<String, String> params = new LinkedHashMap<>();
     return makeGetRequest(BASE_URL + "albums/" + albumId, Optional.of(params), GoogleAlbum.class);
   }
 
-  GoogleMediaItem getMediaItem(String mediaId) throws IOException, InvalidTokenException, PermissionDeniedException {
+  public GoogleMediaItem getMediaItem(String mediaId) throws IOException, InvalidTokenException, PermissionDeniedException {
     Map<String, String> params = new LinkedHashMap<>();
     return makeGetRequest(BASE_URL + "mediaItems/" + mediaId, Optional.of(params), GoogleMediaItem
         .class);
   }
 
-  MediaItemSearchResponse listMediaItems(Optional<String> albumId, Optional<String> pageToken)
+  public MediaItemSearchResponse listMediaItems(Optional<String> albumId, Optional<String> pageToken)
       throws IOException, InvalidTokenException, PermissionDeniedException, UploadErrorException {
     Map<String, Object> params = new LinkedHashMap<>();
     params.put(PAGE_SIZE_KEY, String.valueOf(MEDIA_PAGE_SIZE));