Cancel Job Route

[nickscamara]

closes #106

Notes: Bull doesnt allow to cancel active jobs.

Design decision: we mark the job as failed when someone cancels, and we stop the scraping process (which is the actual expensive process). We bill them as soon as they cancel, even if some promises are not resolved resulting in them getting more from what they paid (totally cool!)

I think this works fairly well for now. @rafaelsideguide would love a look over.

[nickscamara]

I ended up upgrading our fly setup here. Feel free to ignore.

[rafaelsideguide]

Introduced a 500 error on /scrape and /search. Also, the cancel test is not passing.

• E2E Tests for API Routes › POST /v0/scrape › should return a successful response with a valid preview token

  expect(response.statusCode).toBe(200) // Object.is equality

  Expected: 200
  Received: 500

• E2E Tests for API Routes › POST /v0/scrape › should return a successful response with a valid API key

  expect(response.statusCode).toBe(200)

  Expected: 200
  Received: 500

• E2E Tests for API Routes › POST /v0/search › should return a successful response with a valid API key

  expect(response.statusCode).toBe(200);

  Expected: 200
  Received: 500

• E2E Tests for API Routes › If someone cancels a crawl job, it should turn into failed status

  expect(completedResponse.body.partial_data?.length ?? 0).toBeLessThanOrEqual(completedResponse.body.data?.length ?? 0)

  Expected: <= 0
  Received: 40

• E2E Tests for API Routes › POST /v0/scrape with LLM Extraction › should extract data using LLM extraction mode

  expect(response.statusCode).toBe(200) // Object.is equality

  Expected: 200
  Received: 500

[nickscamara]

Thanks @rafaelsideguide, pushed a fix

[nickscamara]

• Add to docs