Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(cli): 1C: PPROF extensions: PPROF dump as PEM #3673

Open
wants to merge 41 commits into
base: master
Choose a base branch
from
Open

feat(cli): 1C: PPROF extensions: PPROF dump as PEM #3673

wants to merge 41 commits into from

Conversation

aaron-kasten
Copy link
Collaborator

This is a continuation of #3454

This is one of 8 PRs in a PR train:
aaron-kasten/kopia:pprof-extensions-c-1a
aaron-kasten/kopia:pprof-extensions-c-1b
aaron-kasten/kopia:pprof-extensions-c-1c
aaron-kasten/kopia:pprof-extensions-c-1d
aaron-kasten/kopia:pprof-extensions-c-1e
aaron-kasten/kopia:pprof-extensions-c-1f
aaron-kasten/kopia:pprof-extensions-c-1g

Usage

pprof dumps are configured using the KOPIA_DEBUG_PPROF environment variable. The variable is a list of pprof profile names (see pprof.Lookup) separated by ,. Optional parameters can be set with '=', delimited by ':'.

example:

export KOPIA_DEBUG_PPROF=cpu,heap=debug=1,mutex=debug=1:rate=1000

The above setting will produce CPU, heap and mutex profiles. The block profile will have its debug parameter set to 1 and its sample rate set to 1000

Once run, profile data will be output in the Kopia logs on termination. Profile dumps are generated as base64 output (PEM) to the log on termination.

You should consider captureing logs to a file when running the Kopia command:

$ kopia --log-file ./myout.log snapshot create . &
[1] 77308

Once the logs are captured, a dump can created by terminating the command:

$ kill %1

The following signals (on Linux and macos) can be used to dump profiles: SIGTERM, SIGINT, and SIGUSR1.

Captured standard-output should look similar to:

saving PEM buffers for output
2021/11/16 19:38:59 Shutting down...
dumping PEM for "PPROF MEM"
-----BEGIN PPROF MEM-----
H4sIAAAAAAAE/7R8CXxURfIw3ZOEJkTTGUUK8Hg8FZMoM8kDBHTXlUtFPBDwWteN
w+RlGJnMG2cmIO7uf4PcN8qtcsqNXCIgghgQBMUL8RYVFPHAAw/UVdTvV9VvzswE
.
.
.
sOBRWRMwE4RCnvDAimrjlBhatdcTCGSqZ9kAbk+k2i4YRBLMVXsCvgQzQXNwKGzh
l3HYSLE3drEgweOQCN7Jjnq8A43CtKt5Rt5tgyoj1u1G/m2DIkMiXk8gcPv/CwAA
//8psEjOrZ4AAA==
-----END PPROF MEM-----

The captured output can then be converted to a pprof binary by using kats. The Kopia kats tool can be used to convert the PEM file into a binary:

$ ./go/bin/kats dump.b64
writing PEM "PPROF MEM" to file "pprof_mem.bin"

When successful, kats will output the file found in the capture file.

kats expects that there is a well formed PEM record in the capture file.

 ./go/bin/kats --help
 Usage of ./go/bin/kats:
  -verbose
    	verbose outout

Once successful, the binary can be used in PPROF:

$ go tool pprof ./pprof_mem.bin
File: pprof_mem.bin
Type: inuse_space
Time: Nov 16, 2021 at 11:38am (PST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) tree
Showing nodes accounting for 35248.89kB, 100% of 35248.89kB total
Showing top 80 nodes out of 127
----------------------------------------------------------+-------------
      flat  flat%   sum%        cum   cum%   calls calls% + context
----------------------------------------------------------+-------------
   16384kB 46.48% 46.48%    16384kB 46.48%                | kopia/tracing.StartProfileBuffers
----------------------------------------------------------+-------------
                                         2561.41kB   100% |   encoding/json.(*decodeState).object
 2561.41kB  7.27% 53.75%  2561.41kB  7.27%                | reflect.mapassign

Configuration Options

TBDThis PR adds the ability to dump pprof data to logs for debugging.

This is a continuation of #3454

This is one of 4 PRs in a PR train:
aaron-kasten/kopia:pprof-extensions-A
aaron-kasten/kopia:pprof-extensions-B
aaron-kasten/kopia:pprof-extensions-C
aaron-kasten/kopia:pprof-extensions-D

Usage

pprof dumps are configured using the KOPIA_DEBUG_PPROF environment variable. The variable is a list of pprof profile names (see pprof.Lookup) separated by ,. Optional parameters can be set with '=', delimited by ':'.

example:

export KOPIA_DEBUG_PPROF=cpu,heap=debug=1,mutex=debug=1:rate=1000

The above setting will produce CPU, heap and mutex profiles. The block profile will have its debug parameter set to 1 and its sample rate set to 1000

Once run, profile data will be output in the Kopia logs on termination. Profile dumps are generated as base64 output (PEM) to the log on termination.

You should consider captureing logs to a file when running the Kopia command:

$ kopia --log-file ./myout.log snapshot create . &
[1] 77308

Once the logs are captured, a dump can created by terminating the command:

$ kill %1

The following signals (on Linux and macos) can be used to dump profiles: SIGTERM, SIGINT, and SIGUSR1.

Captured standard-output should look similar to:

saving PEM buffers for output
2021/11/16 19:38:59 Shutting down...
dumping PEM for "PPROF MEM"
-----BEGIN PPROF MEM-----
H4sIAAAAAAAE/7R8CXxURfIw3ZOEJkTTGUUK8Hg8FZMoM8kDBHTXlUtFPBDwWteN
w+RlGJnMG2cmIO7uf4PcN8qtcsqNXCIgghgQBMUL8RYVFPHAAw/UVdTvV9VvzswE
.
.
.
sOBRWRMwE4RCnvDAimrjlBhatdcTCGSqZ9kAbk+k2i4YRBLMVXsCvgQzQXNwKGzh
l3HYSLE3drEgweOQCN7Jjnq8A43CtKt5Rt5tgyoj1u1G/m2DIkMiXk8gcPv/CwAA
//8psEjOrZ4AAA==
-----END PPROF MEM-----

The captured output can then be converted to a pprof binary by using kats. The Kopia kats tool can be used to convert the PEM file into a binary:

$ ./go/bin/kats dump.b64
writing PEM "PPROF MEM" to file "pprof_mem.bin"

When successful, kats will output the file found in the capture file.

kats expects that there is a well formed PEM record in the capture file.

 ./go/bin/kats --help
 Usage of ./go/bin/kats:
  -verbose
    	verbose outout

Once successful, the binary can be used in PPROF:

$ go tool pprof ./pprof_mem.bin
File: pprof_mem.bin
Type: inuse_space
Time: Nov 16, 2021 at 11:38am (PST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) tree
Showing nodes accounting for 35248.89kB, 100% of 35248.89kB total
Showing top 80 nodes out of 127
----------------------------------------------------------+-------------
      flat  flat%   sum%        cum   cum%   calls calls% + context
----------------------------------------------------------+-------------
   16384kB 46.48% 46.48%    16384kB 46.48%                | kopia/tracing.StartProfileBuffers
----------------------------------------------------------+-------------
                                         2561.41kB   100% |   encoding/json.(*decodeState).object
 2561.41kB  7.27% 53.75%  2561.41kB  7.27%                | reflect.mapassign

Configuration Options

TBD

@aaron-kasten aaron-kasten changed the title Pprof extensions c 1c feat(cli): 1C: PPROF extensions: PPROF dump as PEM Feb 22, 2024
@codecov Codecov
Copy link

codecov bot commented Feb 22, 2024 •
edited

Codecov Report

Attention: Patch coverage is 93.75000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 77.03%. Comparing base (cb455c6) to head (8fa7350).
Report is 55 commits behind head on master.

❗ Current head 8fa7350 differs from pull request most recent head 6034b0e. Consider uploading reports for the commit 6034b0e to get more accurate results

Files Patch % Lines
internal/pproflogging/pproflogging.go 93.75% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #3673      +/-   ##
==========================================
+ Coverage   75.86%   77.03%   +1.17%     
==========================================
  Files         470      471       +1     
  Lines       37301    28618    -8683     
==========================================
- Hits        28299    22047    -6252     
+ Misses       7071     4638    -2433     
- Partials     1931     1933       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

aaron-kasten
aaron-kasten commented Feb 23, 2024
View reviewed changes
internal/pproflogging/pproflogging.go Outdated
buf *bytes.Buffer
var (
// ErrEmptyProfileName returned when a profile configuration flag has no argument.
ErrEmptyProfileName = errors.New("empty profile flag")
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

configurations will be checked - so need some errore

aaron-kasten
aaron-kasten commented Feb 23, 2024
View reviewed changes
internal/pproflogging/pproflogging.go Outdated
ErrEmptyProfileName = errors.New("empty profile flag")

//nolint:gochecknoglobals
pprofConfigs = newProfileConfigs(os.Stderr)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Introduce global config location

aaron-kasten
aaron-kasten commented Feb 23, 2024
View reviewed changes
internal/pproflogging/pproflogging.go Outdated
)

// Writer interface supports destination for PEM output.
type Writer interface {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Writer is a special case. In most cases, writer will be *io.File, but also want to allow more generic types for testing. WriteString support is necessary for proper log output.

aaron-kasten
aaron-kasten commented Feb 23, 2024
View reviewed changes
internal/pproflogging/pproflogging.go Outdated
}

// ProfileConfigs configuration flags for all requested profiles.
type ProfileConfigs struct {
mu sync.Mutex

// +checklocks:mu
wrt Writer
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Output location for PEMs

aaron-kasten
aaron-kasten commented Feb 23, 2024
View reviewed changes
internal/pproflogging/pproflogging.go Outdated
var err error

pprofConfigs.pcm, err = parseProfileConfigs(bufSizeB, ppconfigs)
if err != nil {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested my @julio-lopez

aaron-kasten
aaron-kasten commented Feb 23, 2024
View reviewed changes
internal/pproflogging/pproflogging.go
@@ -237,30 +284,31 @@ func StartProfileBuffers(ctx context.Context) {
}

// DumpPem dump a PEM version of the byte slice, bs, into writer, wrt.
func DumpPem(bs []byte, types string, wrt *os.File) error {
func DumpPem(ctx context.Context, bs []byte, types string, wrt Writer) error {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Context can be used to cancel PEM dump. Unfortunately there's no context aware output in Go :/ - but its a good start.

aaron-kasten and others added 23 commits February 23, 2024 12:05
@aaron-kasten
fixups
5b06ebf
@aaron-kasten
Merge branch 'master' into pprof_extensions-c-1b
e09b963
@aaron-kasten @Shrekster
Update internal/pproflogging/pproflogging.go
8ca44c5 
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
@aaron-kasten @Shrekster
Update internal/pproflogging/pproflogging.go
6411f58 
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
@aaron-kasten
ensure that both ends of the pipe are closed
5977145
@aaron-kasten
Merge branch 'master' of https://github.com/kopia/kopia into pprof_ex…
62cac45 
…tensions-c-1b
@aaron-kasten
Merge branch 'pprof_extensions-c-1b' of ssh://github.com/kastenhq/kop…
c54154d 
…ia into pprof_extensions-c-1b
@aaron-kasten
happy lint
0dd6d26
@aaron-kasten
Merge branch 'master' of https://github.com/kopia/kopia into pprof_ex…
d66d413 
…tensions-c-1b
@aaron-kasten
Merge branch 'pprof_extensions-c-1b' of ssh://github.com/kastenhq/kop…
e442f05 
…ia into pprof_extensions-c-1b
@aaron-kasten
Merge branch 'pprof_extensions-c-1b' of ssh://github.com/kastenhq/kop…
149ef6d 
…ia into pprof_extensions-c-1c
@aaron-kasten @Shrekster
Update internal/pproflogging/pproflogging.go
a2f8f0a 
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
@aaron-kasten
Merge branch 'master' into pprof_extensions-c-1b
2226ee1
@aaron-kasten @Shrekster
Update internal/pproflogging/pproflogging_test.go
5c991da 
Co-authored-by: Shikhar Mall <mall.shikhar.in@gmail.com>
@aaron-kasten
comment for writer
deb5af0
@aaron-kasten
Merge branch 'pprof_extensions-c-1b' of ssh://github.com/kastenhq/kop…
3de9ad5 
…ia into pprof_extensions-c-1b
@aaron-kasten
Merge branch 'pprof_extensions-c-1b' of ssh://github.com/kastenhq/kop…
49149d1 
…ia into pprof_extensions-c-1c
@aaron-kasten
merge c-1b
319362f
@aaron-kasten
Merge branch 'master' of ssh://github.com/kastenhq/kopia into pprof_e…
f5531d1 
…xtensions-c-1c
@aaron-kasten
move context check from D PR
33d6869
@aaron-kasten
better coverage
6cc2a95
@aaron-kasten
Merge branch 'master' into pprof_extensions-c-1c
8fa7350
@aaron-kasten
Merge branch 'master' into pprof_extensions-c-1c
6034b0e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@julio-lopez julio-lopez Awaiting requested review from julio-lopez

@Shrekster Shrekster Awaiting requested review from Shrekster

@jkowalski jkowalski Awaiting requested review from jkowalski

@chaitalisg chaitalisg Awaiting requested review from chaitalisg

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@aaron-kasten