Reduce googleapi calls

[adrinr]

Description

Google api calls are pretty slow, and due the current flow we can have multiple requests per read. The current logic runs as it follows:

1. Get auth token
2. Initialise client
3. Fetch headers
4. Fetch rows
5. If we have as much results as the requested limit, check for next page info, as it follows:
6. Get auth token
7. Initialise client
8. Fetch headers
9. Fetch rows

This contains redundant calls. Storing the client references in the context, we are performing much quicker calls. Results as described below. Both scenarios ran with running a search with pagination results.
Before, avg. of 5.17s:

     checks.....................: 100.00% ✓ 12       ✗ 0  
     data_received..............: 9.8 kB  315 B/s
     data_sent..................: 3.0 kB  96 B/s
     http_req_blocked...........: avg=284µs   min=2µs   med=7µs    max=1.67ms  p(90)=841µs   p(95)=1.25ms  
     http_req_connecting........: avg=57.16µs min=0s    med=0s     max=343µs   p(90)=171.5µs p(95)=257.24µs
     http_req_duration..........: avg=5.16s   min=4.03s med=5.18s  max=6.29s   p(90)=5.98s   p(95)=6.13s   
     http_req_failed............: 100.00% ✓ 6        ✗ 0  
     http_req_receiving.........: avg=2.08ms  min=134µs med=339µs  max=10.82ms p(90)=5.74ms  p(95)=8.28ms  
     http_req_sending...........: avg=35.66µs min=14µs  med=37.5µs max=54µs    p(90)=50µs    p(95)=52µs    
     http_req_tls_handshaking...: avg=0s      min=0s    med=0s     max=0s      p(90)=0s      p(95)=0s      
     http_req_waiting...........: avg=5.16s   min=4.03s med=5.18s  max=6.29s   p(90)=5.97s   p(95)=6.13s   
     http_reqs..................: 6       0.193294/s
     iteration_duration.........: avg=5.17s   min=4.03s med=5.18s  max=6.29s   p(90)=5.99s   p(95)=6.14s   
     iterations.................: 6       0.193294/s
     vus........................: 1       min=1      max=1
     vus_max....................: 1       min=1      max=1


running (0m31.0s), 0/1 VUs, 6 complete and 0 interrupted iterations

After, avg. of 2.05s:

     checks.....................: 100.00% ✓ 30       ✗ 0  
     data_received..............: 24 kB   792 B/s
     data_sent..................: 7.5 kB  242 B/s
     http_req_blocked...........: avg=117.19µs min=2µs  med=2µs   max=1.72ms p(90)=4.8µs   p(95)=520.49µs
     http_req_connecting........: avg=23µs     min=0s   med=0s    max=345µs  p(90)=0s      p(95)=103.49µs
     http_req_duration..........: avg=2.05s    min=1.1s med=2.05s max=3.24s  p(90)=2.62s   p(95)=2.84s   
     http_req_failed............: 100.00% ✓ 15       ✗ 0  
     http_req_receiving.........: avg=334.66µs min=89µs med=186µs max=1.47ms p(90)=941.6µs p(95)=1.39ms  
     http_req_sending...........: avg=25.33µs  min=13µs med=18µs  max=80µs   p(90)=49.4µs  p(95)=68.09µs 
     http_req_tls_handshaking...: avg=0s       min=0s   med=0s    max=0s     p(90)=0s      p(95)=0s      
     http_req_waiting...........: avg=2.05s    min=1.1s med=2.05s max=3.24s  p(90)=2.62s   p(95)=2.84s   
     http_reqs..................: 15      0.486808/s
     iteration_duration.........: avg=2.05s    min=1.1s med=2.05s max=3.24s  p(90)=2.62s   p(95)=2.84s   
     iterations.................: 15      0.486808/s
     vus........................: 1       min=1      max=1
     vus_max....................: 1       min=1      max=1


running (0m30.8s), 0/1 VUs, 15 complete and 0 interrupted iterations

Addresses

• BUDI-8248 - Improve Google Sheets api usage

Launchcontrol

Improving google sheets read performance

Feature branch env

Feature Branch Link

[linear]

BUDI-8248 Improve Google Sheets api usage

[adrinr]

As this is temporal, I use a package override. This will help us as we will not need to update code references

[mike12345567]

very nice - good way to solve this!