-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
POC: Speed up compilation by freezing container during compilation. #11940
base: master
Are you sure you want to change the base?
Conversation
Instead of rebuilding the entire project every time we want to compile a single fuzz target, a better workflow is to build the project once and somehow compile the target against the already compiled project code. This POC does that by interrupting building in when it detects it is compiling the fuzz target. On detection it does the following: 1. Writes the command to /out/statefile TODO: write the cwd. 2. Commits the current container as "frozen" for use later. TODO: make this changeable. 3. Returns 1 so compilation stops. TODO: It would be better to exit the container. This step may be important to prevent clean up of the environment. Then the frozen container can be used to compile fuzz targets against the project without recompiling the project in its entirety. TODO: 1. Support this in oss-fuzz-gen 2. Install docker command line tool in base-builder (or use sneaky inheritance) because it must be used within the container.
Command to test:
|
|
||
func IsCompilingTarget(cmdline []string) bool { | ||
for _, arg := range cmdline { | ||
// This can fail if people do crazy things they aren't supposed to e.g. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this sentence complete or something missing?
} | ||
|
||
func WriteTargetArgsAndCommitImage(cmdline []string) { | ||
fmt.Println("WRITE COMMAND") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shall we remove the debug print, or make it more informative?
E.g., 'JCC WRITE COMMAND: <compilation command>'
.
|
||
func WriteTargetArgsAndCommitImage(cmdline []string) { | ||
fmt.Println("WRITE COMMAND") | ||
f, _ := os.OpenFile("/out/statefile.json", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IIRC, os.O_APPEND
caused an invalid JSON format during our hackathon, is it the same case here?
Instead of rebuilding the entire project every time we want to compile a single fuzz target, a better workflow is to build the project once and somehow compile the target against the already compiled project code.
This POC does that by interrupting building in when it detects it is compiling the fuzz target.
On detection it does the following:
This step may be important to prevent clean up of the environment.
Then the frozen container can be used to compile fuzz targets against the project without recompiling the project in its entirety.
TODO: