Skip to content

fix: Do not publish .proto files to npm#8108

Draft
danieljbruce wants to merge 4 commits intomainfrom
protos-do-not-publish
Draft

fix: Do not publish .proto files to npm#8108
danieljbruce wants to merge 4 commits intomainfrom
protos-do-not-publish

Conversation

@danieljbruce
Copy link
Copy Markdown
Contributor

Description

This pull request addresses issue googleapis/librarian#4820. The .proto files should be the only files removed from the build so that customers enjoy reduced package dependency size.

Note that we also change npm run compile so that build/protos doesn't get created. This is advantageous because it ensures that when testing we will discover that the build fails earlier when the tests don't work instead of the build failing when we try to publish the package.

Impact

Will speed up npm install for customers and reduce customer burden.

Testing

It is worth calling out that it is kind of difficult to test package publishing. We could cut a pre-release and see what happens, but maybe this isn't necessary since npm will stop publishing if any changes break the build anyway. We also considered the impact of customers reaching into the library to reference the .proto files, but these files are unlikely to be useful to customers anyway.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates numerous package.json files to exclude .proto files from published npm packages and introduces a command to delete the build/protos/google directory during compilation. The review feedback correctly identifies a critical issue: explicitly deleting the google directory removes essential JSON descriptors required by the libraries at runtime, which would break the generated clients. Since the .proto files are already excluded via the files array in package.json, the reviewer recommends removing the destructive rm -rf commands from the compile and prepare scripts to ensure library stability.

"lint": "gts check src samples test",
"clean": "gts clean",
"compile": "tsc -p . && cp src/*.json build/src && cp -r test/fixtures build/test && cp -r protos build/",
"compile": "tsc -p . && cp src/*.json build/src && cp -r test/fixtures build/test && cp -r protos build/ && rm -rf build/protos/google",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The command rm -rf build/protos/google is too broad. It deletes the entire google directory tree within build/protos, which typically contains not only .proto files but also the .json descriptors required by the library at runtime (e.g., google/api/http.json, google/longrunning/operations.json). Deleting these will break the library's ability to load these descriptors. Since you have already added !build/protos/**/*.proto to the files array in package.json, the .proto files will be excluded from the published npm package automatically. If the intention of deleting them during compile is to ensure tests don't rely on them, you should specifically target .proto files while preserving .json files.

Suggested change
"compile": "tsc -p . && cp src/*.json build/src && cp -r test/fixtures build/test && cp -r protos build/ && rm -rf build/protos/google",
"compile": "tsc -p . && cp src/*.json build/src && cp -r test/fixtures build/test && cp -r protos build/",

"compile-http-protos": "pbjs -t static-module -r http_proto --keep-case google/api/http.proto -p ./protos > protos/http.js && pbts protos/http.js -o protos/http.d.ts",
"fix": "gts fix",
"prepare": "npm run compile && cd ../tools && npm i && npm run compile && cd ../gax && pwd && node ../tools/build/src/prepublish.js . && mkdirp build/protos && cp -r protos/* build/protos/ && npm run minify-proto-json",
"prepare": "npm run compile && cd ../tools && npm i && npm run compile && cd ../gax && pwd && node ../tools/build/src/prepublish.js . && mkdirp build/protos && cp -r protos/* build/protos/ && rm -rf build/protos/google && npm run minify-proto-json",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

As noted in the compile script, rm -rf build/protos/google will delete required JSON descriptors. Additionally, this prepare script redundantly copies protos to build/protos after npm run compile has already done so. It is safer to rely on the files exclusion in package.json to prevent publishing .proto files while preserving the necessary JSON files.

Suggested change
"prepare": "npm run compile && cd ../tools && npm i && npm run compile && cd ../gax && pwd && node ../tools/build/src/prepublish.js . && mkdirp build/protos && cp -r protos/* build/protos/ && rm -rf build/protos/google && npm run minify-proto-json",
"prepare": "npm run compile && cd ../tools && npm i && npm run compile && cd ../gax && pwd && node ../tools/build/src/prepublish.js . && mkdirp build/protos && cp -r protos/* build/protos/ && npm run minify-proto-json",

@@ -25,7 +26,7 @@
],
"scripts": {
"clean": "gts clean",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Deleting build/protos/google will remove the package's own proto descriptors (e.g., google/cloud/<service>/v1/<service>.json) if they are located within that directory structure. This will break the generated client at runtime. The exclusion of .proto files from the npm package is already correctly handled by the files array entry !build/protos/**/*.proto. You should remove the rm -rf command to ensure the necessary JSON descriptors are preserved in the build.

Suggested change
"clean": "gts clean",
"compile": "tsc -p . && cp -r protos build/{% if not api.legacyProtoLoad %} && minifyProtoJson{% endif %}",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I notice this change is inside core. I'm somewhat concerned about that.

"babel": "babel esm --out-dir build/cjs --ignore \"esm/**/*.d.ts\" --extensions \".ts\" --out-file-extension .cjs --copy-files",
"compile:cjs": "tsc -p ./tsconfig.json && npm run babel",
"compile": "npm run compile:esm && rm -rf esm/src/json-helper.d.cts && npm run compile:cjs && rm -rf build/protos && cp -r protos build/protos",
"compile": "npm run compile:esm && rm -rf esm/src/json-helper.d.cts && npm run compile:cjs && rm -rf build/protos && cp -r protos build/protos && rm -rf build/protos/google",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

As with the CJS template, rm -rf build/protos/google is likely to delete required JSON descriptors, breaking the library. The files array already excludes .proto files from the package. Recommend removing this deletion step from the compile script.

Suggested change
"compile": "npm run compile:esm && rm -rf esm/src/json-helper.d.cts && npm run compile:cjs && rm -rf build/protos && cp -r protos build/protos && rm -rf build/protos/google",
"compile": "npm run compile:esm && rm -rf esm/src/json-helper.d.cts && npm run compile:cjs && rm -rf build/protos && cp -r protos build/protos",

],
"scripts": {
"compile": "tsc -p . && cp -r proto* build/ && cp -r testproxy/proto* build/testproxy/",
"compile": "tsc -p . && cp -r proto* build/ && rm -rf build/protos/google && cp -r testproxy/proto* build/testproxy/",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Deleting build/protos/google will remove the Bigtable proto descriptors, breaking the library. The files array already handles the exclusion of .proto files from the published package.

Suggested change
"compile": "tsc -p . && cp -r proto* build/ && rm -rf build/protos/google && cp -r testproxy/proto* build/testproxy/",
"compile": "tsc -p . && cp -r proto* build/ && cp -r testproxy/proto* build/testproxy/",

Copy link
Copy Markdown
Contributor Author

@danieljbruce danieljbruce left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should address this item

@@ -25,7 +26,7 @@
],
"scripts": {
"clean": "gts clean",
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I notice this change is inside core. I'm somewhat concerned about that.

Copy link
Copy Markdown
Contributor Author

@danieljbruce danieljbruce left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to make sure prompting does not change the string that is fed into protos.

"scripts": {
"clean": "gts clean",
"compile": "tsc -p . && cp -r protos build/{% if not api.legacyProtoLoad %} && minifyProtoJson{% endif %}",
"compile": "tsc -p . && cp -r protos build/ && rm -rf build/protos/google{% if not api.legacyProtoLoad %} && minifyProtoJson{% endif %}",
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This generated code here is concerning. The string for the protos command should not change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant