New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
build.zig: issue with caching with chained build.step.Run steps #19817
Comments
Option (2) (not including the full input path in its hash) seems like it would be backwards incompatible to me: Option (1) (moving generated files to a path based on their contents) seems like a more general solution - Just FYI, I think providing a file's content to a run step via |
Unfortunately |
fixes ziglang#19817 This improves the efficiency of the cache when chaining muliple commands like const step1 = b.addRunArtifact(tool_fast); step1.addFileArg(b.path("src/input.c")); const output1 = step1.addOutputFileArg("output1.h"); const step2 = b.addRunArtifact(tool_slow); step2.addFileArg(output2); const chained_output = step2.addOutputFileArg("output2.h"); assume that step2 takes much long time than step1 if we make a change to "src/input.c" which produces an identical "output1.h" as a previous input, one would expect step2 not to rerun as the cached output2.h only depends on the content of output1.h However, this does not work yet as the hash of src/input.c leaks into the file name of the cached output1.h, which the second run step interprets as a different cache key. Erasing the "zig-build/o/{HASH}" part of the file name in the hash key fixes this.
fixes ziglang#19817 This improves the efficiency of the cache when chaining muliple commands like const step1 = b.addRunArtifact(tool_fast); step1.addFileArg(b.path("src/input.c")); const output1 = step1.addOutputFileArg("output1.h"); const step2 = b.addRunArtifact(tool_slow); step2.addFileArg(output1); const chained_output = step2.addOutputFileArg("output2.h"); assume that step2 takes much long time than step1 if we make a change to "src/input.c" which produces an identical "output1.h" as a previous input, one would expect step2 not to rerun as the cached output2.h only depends on the content of output1.h However, this does not work yet as the hash of src/input.c leaks into the file name of the cached output1.h, which the second run step interprets as a different cache key. Erasing the "zig-build/o/{HASH}" part of the file name in the hash key fixes this.
fixes ziglang#19817 This improves the efficiency of the cache when chaining muliple commands like const step1 = b.addRunArtifact(tool_fast); step1.addFileArg(b.path("src/input.c")); const output1 = step1.addOutputFileArg("output1.h"); const step2 = b.addRunArtifact(tool_slow); step2.addFileArg(output1); const chained_output = step2.addOutputFileArg("output2.h"); assume that step2 takes much long time than step1 if we make a change to "src/input.c" which produces an identical "output1.h" as a previous input, one would expect step2 not to rerun as the cached output2.h only depends on the content of output1.h However, this does not work yet as the hash of src/input.c leaks into the file name of the cached output1.h, which the second run step interprets as a different cache key. Erasing the "zig-build/o/{HASH}" part of the file name in the hash key fixes this.
fixes ziglang#19817 This improves the efficiency of the cache when chaining muliple commands like const step1 = b.addRunArtifact(tool_fast); step1.addFileArg(b.path("src/input.c")); const output1 = step1.addOutputFileArg("output1.h"); const step2 = b.addRunArtifact(tool_slow); step2.addFileArg(output1); const chained_output = step2.addOutputFileArg("output2.h"); assume that step2 takes much long time than step1 if we make a change to "src/input.c" which produces an identical "output1.h" as a previous input, one would expect step2 not to rerun as the cached output2.h only depends on the content of output1.h However, this does not work yet as the hash of src/input.c leaks into the file name of the cached output1.h, which the second run step interprets as a different cache key. Erasing the "zig-build/o/{HASH}" part of the file name in the hash key fixes this.
fixes ziglang#19817 This improves the efficiency of the cache when chaining muliple commands like const step1 = b.addRunArtifact(tool_fast); step1.addFileArg(b.path("src/input.c")); const output1 = step1.addOutputFileArg("output1.h"); const step2 = b.addRunArtifact(tool_slow); step2.addFileArg(output1); const chained_output = step2.addOutputFileArg("output2.h"); assume that step2 takes much long time than step1 if we make a change to "src/input.c" which produces an identical "output1.h" as a previous input, one would expect step2 not to rerun as the cached output2.h only depends on the content of output1.h However, this does not work yet as the hash of src/input.c leaks into the file name of the cached output1.h, which the second run step interprets as a different cache key. Erasing the "zig-build/o/{HASH}" part of the file name in the hash key fixes this.
fixes ziglang#19817 This improves the efficiency of the cache when chaining muliple commands like const step1 = b.addRunArtifact(tool_fast); step1.addFileArg(b.path("src/input.c")); const output1 = step1.addOutputFileArg("output1.h"); const step2 = b.addRunArtifact(tool_slow); step2.addFileArg(output1); const chained_output = step2.addOutputFileArg("output2.h"); assume that step2 takes much long time than step1 if we make a change to "src/input.c" which produces an identical "output1.h" as a previous input, one would expect step2 not to rerun as the cached output2.h only depends on the content of output1.h However, this does not work yet as the hash of src/input.c leaks into the file name of the cached output1.h, which the second run step interprets as a different cache key. Erasing the "zig-build/o/{HASH}" part of the file name in the hash key fixes this.
Zig Version
0.13.0-dev.56+956f53beb
Steps to Reproduce and Observed Behavior
Consider build.zig logic where one
build.step.Run
step takes the output of another such step as an input:Assume that step2 is much more expensive than step1. If
src/foo.c
changed we obviously need to re-run step1, but step2 should only need to re-run if thefoo_header
actually changed as a result.But any new state of
src/foo.c
causes both step1 and step2 to run, even whenfoo.generated.h
output is identical. This is because the complete path of the builtfoo_header
depend on the input state of step1. and in turn, the hash calculated bybuild.step.Run.make
in step2 depends not only of the actual contents ofaddFileArg()
file, but also its name which leaks the hash of step1 inputs.Complete executable test case: https://github.com/bfredl/zig-run4run . The first step extracts only the prototypes of the c functions, so changing
implementation_of_foo()
toimplementation_of_foo2()
would illustrate the behavior.Expected Behavior
When doing an incremental rebuild, if a change to
src/foo.c
causes step1 to run but still producing the same"foo.generated.h"
contents as an earlier rebuild, do not run step2 but use the exiting cachedfoo.rpc_bindings.h
from that earlier build.I think this could be done in two different ways:
foo_header.getPath()
to only depend on that file's contents, not all inputs to step1zig-cache/o/{HASH}/foo.generated.h
)I experimented a bit with the second option but I couldn't make it work myself.
The text was updated successfully, but these errors were encountered: