feat: Restructure AIPs 213 and 215, with more common proto details (#1117)

jskeet · web-flow · commit 7a3a67a0fb30 · 2023-06-27T15:31:18.000Z
diff --git a/aip/general/0213.md b/aip/general/0213.md
@@ -9,44 +9,75 @@ placement:
 
 # Common components
 
-In general, API reviewers encourage API producers to keep their APIs mostly
-self-contained, except for a relatively small set of common protos which are
-safe to import (e.g. [`google.protobuf.Timestamp`][timestamp]). This is for
-good reason: APIs generally need to be able to move forward independently of
-one another, and mutual dependencies can cause downstream APIs to be forced
-into taking major version changes or even lead to dependency conflicts.
+As specified in [AIP-215][], APIs must be self-contained except for the use
+of "common component" packages which are intended for use by multiple APIs.
 
-However, there are also cases where common structures are valuable, especially
-where a concept is well-known and it is sufficiently clear that it will not
-change. Common protos serve this use case.
+There are two kinds of common component packages:
 
-## Guidance
+- Organization-specific common components, covering organization-specific
+  concepts such as a "viewport" in Maps.
+- Global common components which are generic (i.e. not domain-specific),
+  such as "timestamp" or "postal address".
 
-The public representation of APIs **should** be self-contained, meaning that
-all protos used by the API originate in the same proto `package`, except for
-common protos, which **may** be used freely in any API.
+Where it is safe to share a single representation across multiple APIs,
+common components can make it easier for clients to interact with those APIs.
+Concept-specific client code can be written once, and messages can be used from
+the response of one API in the request of another without clunky copying, for
+example.
 
-APIs **must not** define a set of API-specific common protos which live outside
-of its versioning structure. This prevents independent movement of particular
-versions and also causes problems for client libraries in many languages that
-compile the proto messages into classes.
+This benefit comes with significant restrictions and limitations, however,
+and should not be attempted lightly.
 
-APIs **should not** directly depend on protos defined in other APIs. Instead,
-they should copy and paste the applicable messages into their own API. When
-doing so, APIs **should** keep the field names and numbers the same.
+Note that even if the *domain* of a component is common, the requirements of
+a component may be organization-specific. For example, some organizations may
+have particular requirements of how financial values are represented, leading
+to multiple finance-oriented organization-specific common components - because
+any global common component would either not meet the organization-specific
+requirements, or be too complex for general use.
 
-## Existing common protos
+## Guidance
 
-The common protos, which public-facing protos for an API **may** safely import,
-are as follows:
+- Organization-wide common component packages **must** end with `.type`, e.g.
+  `google.geo.type` or `google.shopping.type`.
+- Organizations **must** consult the API design team before creating a new
+  organization-wide common component package.
+- Organizations **must not** define generic components in organization-specific
+  common component packages, instead preferring global common components.
+- Common components **must not** be "moved" (that is, deleted from one common
+  component package and added to a different one) from an organization-specific
+  common component package to a global common component package or vice versa.
+  - A common component **may** be copied from an organization-specific common
+    component package to a global common component package (without deleting the
+    original component) if it is found to be more widely-applicable than
+    originally expected.
+- Fields **should not** be added to existing messages.
+- Values **should not** be added to existing enums.
+- Fields **must not** be removed from existing messages.
+- Values **must not** be removed from existing enums.
+- While documentation **may** be clarified, it **should not** change the
+  meanings of existing values, including the validity of any given message
+  or set of messages.
+- New proto messages and enums **may** be added to common component packages.
+  - API teams  **should** allow sufficient time for propagation to clients
+    before using the new messages and enums in their APIs. Fields may take some
+    time for any changes to propagate through publication to client libraries
+    and other surfaces.
+  - API teams **should** consult widely within their organization, and ideally
+    with the API design team, before adding a new message or enum, due to the
+    limitations listed above.
+
+## Existing global common components
+
+The global common components, which public-facing protos for an API **may** safely
+import, are as follows:
 
 - [`google.api.*`](https://github.com/googleapis/googleapis/blob/master/google/api) (but *not* subpackages of `google.api`)
 - [`google.longrunning.Operation`](https://github.com/googleapis/googleapis/blob/master/google/longrunning/operations.proto)
 - [`google.protobuf.*`](https://github.com/protocolbuffers/protobuf/tree/master/src/google/protobuf)
 - [`google.rpc.*`](https://github.com/googleapis/googleapis/blob/master/google/rpc/)
 - [`google.type.*`][type]
 
-Note that some common protos may have internal-only fields. APIs **should**
+Note that some common components may have internal-only fields. APIs **should**
 generally only rely on fields which have been
 [released into open source](https://github.com/googleapis/googleapis).
 
@@ -56,9 +87,9 @@ IAM messages used throughout Google.
 <!-- prettier-ignore -->
 [iam]: https://github.com/googleapis/googleapis/tree/master/google/iam/v1
 
-**Note:** Many APIs also import protos from other packages for internal-only
+**Note:** Many APIs also import components from other packages for internal-only
 use (e.g. to apply visibility labels or provide instructions to internal
-infrastructure). This is acceptable provided that the _public_ protos do not
+infrastructure). This is acceptable provided that the _public_ components do not
 contain such references.
 
 ### Protobuf types
@@ -124,31 +155,32 @@ and the definitive list is always [the code][type], several types deserve note:
 
 Occasionally, it may be useful to add protos to these packages or to add to the
 list of commonly-available protos. In order to do this, [open an issue][] on
-the AIP repository in GitHub.
-
-However, some general guidelines are worth noting for this:
-
-- Protos **should** only be promoted to common status if we are certain that
-  they will never change (at all -- even in ways that would normally be
-  considered backwards compatible). Common protos are generally not versioned,
-  and it must be the case that we can rely on the proto to be a complete and
-  accurate representation indefinitely.
-  - The exception to this is protos describing our infrastructure, which
-    **may** have rare, backwards-compatible changes.
-- Protos must be applicable to a significant number of APIs for consideration
-  as common protos. It is okay for those APIs to be clustered together (e.g.
-  all in a single PA).
-- There is no good way to "stage" a common proto, because moving references to
-  them is effectively not possible. (In other words, it is infeasible to add a
-  proto to `google.geo.type.*` and then "graduate" it to `google.type.*`
-  later.)
-- Adding a common proto requires coordination between several teams, and it may
-  take time between when an addition is approved and when it is available for
-  use.
-- Even after a common proto is added, APIs using local versions must continue
-  to do so until they go to the next major version.
-
-In the event that you believe adding a common proto is appropriate, please
-[open an issue][].
+the AIP repository in GitHub, noting the guidelines above.
+
+## Rationale
+
+Common components are effectively unversioned: APIs evolve independently of
+each other, both in terms of definition and implementation. A change such as
+adding a field is backward-compatible and predictable in specific APIs, and the
+API team can ensure that the server implementation is available before the API
+definition is published. By contrast, a change in a common component would
+effectively be universally available even if most API implementations did not
+take it into account.
+
+Adding a new message or enum is backward-compatible, as it does not affect
+existing APIs that may import other messages or enums from the same common
+component package.
+
+Consultation with the API design team is required for global common components
+and suggested for organization-specific common components as the border between
+"generic" and "organization-specific" is a gray area; some generic *concepts*
+have organization-specific use cases which surface through the components.
+
+## Changelog
+
+- 2023-06-XX: Restructured AIPs 215 and 213 for clarity, and introduced the
+  concept of organization-wide common protos more formally.
+- 2018-08-17: Initial AIP written.
 
 [open an issue]: https://github.com/googleapis/aip/issues
+[aip-215]: ./0215.md
diff --git a/aip/general/0215.md b/aip/general/0215.md
@@ -7,59 +7,57 @@ placement:
   order: 30
 ---
 
-# Common component versions
+# API-specific protos
 
-Many APIs may support more than one version at the same time. Often, our first
-instinct is to create protos which are intended to be shared between API
-versions, and place them in an unversioned "common" directory. A similar
-variant to this is an _omitted_ version: a message or service that is
-_implicitly_ v1, but has no version in the proto package.
-
-When protos are unversioned, changing them safely (that is, in a backwards
-compatible way) is very difficult. For example, adding a field to an
-unversioned proto effectively adds the field to all _existing_ versions, which
-do not actually support it. This is surprising to users, and creates a
-situation where the proto files are not accurate representations of the API
-surface.
-
-Additionally, client library generators usually generate a class for each
-message in your protos in a language-appropriate namespace. In several
-languages, each version of your API is shipped as a _separate_ client library,
-and the code generator needs to generate the common messages for your API in
-order to ensure a complete package. This makes it difficult to use multiple
-APIs with similar dependent messages together.
-
-For omitted versions, the version is effectively hidden, and it becomes more
-difficult to reason about release phases (alpha, beta, GA), with no substantive
-benefit.
+APIs are mostly defined in terms of protos which are API-specific, with
+occasional dependencies on common components. Keeping APIs isolated from each
+other avoids versioning problems and client library packaging problems.
 
 ## Guidance
 
-- All protos specific to an API **should** be within the versioned package
-  (e.g., `yourapi.v1.SharedProtoMessage`).
-
-- In scenarios where an API doesn't consider itself to have a version, the API
-  **must** use `v1`. (Omitted-version protos are prohibited.)
-
-- When a shared proto is identical, that proto **should** be duplicated to the
-  other versioned package (e.g., copied and pasted into
-  `yourapi.v2.SharedProtoMessage`).
-
-### What if a proto will never change?
-
-There are some situations where it is useful to have an unversioned
-proto. These should generally apply to a complete organization or suite
-of APIs, not just a single API. In these situations, it may be possible
-to add it to our collections of common protos. Common protos are able to
-be used by many APIs, and are always unversioned.
-
-Common protos shall always have a package structure ending in `type` (e.g.
-`google.type` or `google.cloud.type`).
-
-**Warning:** This is a relatively rare occurrence. If you find yourself
-adding protos to your organizations's `type` directory frequently,
-double-check that this is actually where they belong.
-
-For more information on common protos, consult [AIP-213][].
+- All protos specific to an API **must** be within a package with a major version
+  (e.g., `google.library.v1`).
+- References to resources in other APIs **must** be expressed in terms of
+  resource names ([AIP-122][]), rather than using the resource messages.
+- When two versions of an API use effectively the same (API-specific) proto
+  that proto **must** be duplicated in each version. (In other words, APIs
+  **must not** create their own "API-specific common component" packages.)
+- Organization-specific common components **may** be placed in a common package,
+  as described in [AIP-213][], but **must not** be used by any API outside
+  that organization.
+- Global common components (also described in AIP-213) **may** be freely used by any API.
+
+## Rationale
+
+When one API depends on protos defined by another API, this introduces uncertainty
+in terms of customer-expected behavior and client library dependency management.
+Suppose `google.cloud.library.v1` depends on the protos (rather than abstract resources)
+in `google.cloud.movies.v2`. Any change to `google.cloud.movies.v2` can cause problems.
+
+For example:
+
+- If a field is added to a message in `google.cloud.movies.v2`, should customers using
+  `google.cloud.library.v1` expect to see it? If so, how soon after the field has
+  been added? What about other API changes?
+- If the whole major version `google.cloud.movies.v2` is deprecated
+  (typically after v3 has been released), does that mean `google.cloud.library.v1` has
+  to change to use `google.cloud.movies.v3`, and if so, does that require a new major version
+  for the library API as well?
+- How should client library versioning reflect changes to dependent APIs?
+
+Keeping APIs isolated from each other, with a limited set of common components which are
+maintained in a highly disciplined way, reduces a lot of the issues with dependencies.
+
+API-specific common components shared across versions add complexity for client
+library generation and packaging, and are inflexible in terms of versioning.
+When protos are duplicated because they *start* off the same in multiple versions,
+they can still diverge over time as they are isolated from each other.
+
+## Changelog
+
+2023-06-XX: Restructured AIPs 215 and 213 for clarity.
+2023-05-11: Changed "PA" to "organization".
+2018-10-01: Initial AIP written.
 
 [aip-213]: ./0213.md
+[aip-122]: ./0122.md
