Add ability to run JS scripts during archiving with Playwright/Puppeteer

[pirate]

https://github.com/GoogleChrome/puppeteer is fantastic for scripting actions on pages before making a screenshot or PDF.

I could add support for custom puppeteer scripts for certain urls that need a user action to be performed before archiving (e.g. logging in or closing a welcome message popup).

Puppeteer code looks like this:

        const browser = await puppeteer.launch({headless: false})
        const page = await browser.newPage()

        await page.goto('https://carbon.now.sh')

        const code_input = 'div.ReactCodeMirror div.CodeMirror-code > pre:nth-child(11)'
        await page.click(code_input)
        await page.keyboard.down('Meta')
        await page.keyboard.down('a')
        await page.keyboard.up('a')
        await page.keyboard.up('Meta')
        await page.keyboard.press('Backspace')

[pirate]

Would go well with: https://github.com/checkly/puppeteer-recorder

[FiloSottile]

archive.is has a nice set of scripts that do things like expanding all Reddit threads or scrolling through Twitter timelines before taking a snapshot. This is the kind of thing I've seen develop a nice community around with youtube-dl.

[pirate]

The beginnings of this will start to be implemented with our move from chromium-browser to pyppeteer playwright: #177, then these will be possible:

• support for scripted user flows (this ticket)
• dismissing gdpr / cookie / subscription / donation popups automatically: Dismiss cookie permission, GDPR, and subscription popups before archiving #175
• autoscroll before archiving with full-page dynamic height screenshots: Autoscroll before before archiving and take full-height screenshots #80
• dynamic/interactive requests saving into the WARC with pypetteer running through pywb: Extend WARC file with all requests made via all archive methods #130

[n0ncetonic]

I have experience with coding Puppeteer scripts and I'm willing to start either implementing fixes for #175 #80 #130 as independent code samples in preparation for pyppeteer or to start a branch that just replicates current functionality but with pyppeteer depending on whether or not you've started a private branch or prefer to implement it yourself

[pirate]

Sweet, the super-rough planned design is for ArchiveBox to run user-provided scripts like this:

archive_scripts = {
    'dismiss_modals: '() => {document.querySelectorAll(".modal").delete()}',
    ...
}


browser = await launch()
page = await browser.newPage()

for link in links:
    await page.goto(link['url'])

    for script_name, script_js in archive_scripts:
        link['history'][script_name].append(await page.evaluate(script_js))

    link['history']['screenshot'].append(await page.screenshot({'path': 'screenshot.png'}))
    link['history']['pdf'].append(await page.print_pdf({'path': 'output.pdf'}))

await browser.close()

The final implementation will be more fully-featured than this of course. S context and any output returned gets saved as an ArchiveResult entry like any other extractor.

[n0ncetonic]

Alright cool, I will start working on getting that implemented on my fork.

Planning to do this in 3 phases across two milestones which I think align well with the current roadmap.

Phase I. import pyppeteer and replace all current chromium-browser calls with pyppeteer equivalents.

Milestone I. ArchiveBox migrated to pyppeteer

Phase II. Implement minimalist scripting support allowing users to extend browser-based modules using javascript.

Milestone II. Codebase aligned with Roadmap's Long Term Change to allow user-defined scripting of the browser.

Phase III. Bootstrap collection of browser scripts by creating and including

• autoscroll_screenshot.js - screenshot capturing of the entire page by autoscrolling Autoscroll before before archiving and take full-height screenshots #80
• anti_detection.js - bypasses detection/blocking of headless browser via selective overwriting of page-wide getter properties this is something I have working for a personal project that leveraged Puppeteer
• cookie_accept.js - generic enumeration and dismissal of GDPR/subscription/cookie popups Dismiss cookie permission, GDPR, and subscription popups before archiving #175

Note: As my primary aim will be to make progress on the Roadmap #130 will not be a requisite for Phase III completion. Once Phase III is complete and merged into master a separate Pull Request will address extending WARC generation.

We'll go to next steps (like mimicking archive_methods.py loading of scripts) after Phase III provides a working, basic scripting subsystem

[pirate]

If possible, work on the Phase III scripts first. Those would be most helpful to me, as I've already started work on the phase I and II steps you outlined above over the last few months.

You can test your scripts using the pyppeteer demo code from their README, and I'll make sure the ArchiveBox API is compatible to work with them.

[pirate]

I found some huge repositories of Seleneium/Puppeteer scripts for dismissing modals and logging in to lots of sites. These are going to be super useful:

• https://github.com/CriseLYJ/awesome-python-login-model
• https://github.com/facert/awesome-spider
• https://github.com/duyetdev/awesome-web-scraper
• https://github.com/BruceDone/awesome-crawler

[pirate]

Whoops closed/reopened by accident. A quick update for those following this issue, we have a number of blocking tasks before we're going to get around to this:

1. Finish refactoring extractors into independent plugin-style modules that define their own config and dependencies
2. Refactor ArchiveBox to use a message-passing/event-sourcing architecture so that all tasks are handled by workers listening on queues
3. Create a playwright/puppeteer message queue worker to handle all headless-browser related tasks in a single browser instance (to avoid launching and closing a browser for each URL/extractor run)
4. Define a spec for user-contributed playwright scripts that are callable by the playwright worker during archiving

Lots of work has been done so far to get us to step 1, but we're still at the foothills of what will be required before this feature is ready for prime-time. It's still high up on our list of desired features but don't expect it anytime soon.

[UmutAlihan]

Looking very forward for these feautures to be implemented. Since Cloudflare now effectively blocks content crawling for many many adopting sites (such as Medium), functionality of Archivebox has a risk to be limited only to a sites which are not yet utilizing Cloudflare's bot detection systems.

(p.s.: Many sites are transitioning to Cloudflare's or similar services. There it is currently very likely to crawl an empty/failing archive like below example.)

[GlassedSilver]

Looking very forward for these feautures to be implemented. Since Cloudflare now effectively blocks content crawling for many many adopting sites (such as Medium), functionality of Archivebox has a risk to be limited only to a sites which are not yet utilizing Cloudflare's bot detection systems.

(p.s.: Many sites are transitioning to Cloudflare's or similar services. There it is currently very likely to crawl an empty/failing archive like below example.)

[snipped image]

+1 to that.

Really, REALLY annoying to see Cloudflare being so overly aggressive.

Like sure, you can be hammering down on me if I try to pull pages from sites protected by your network in the hundreds per minute, fine. I get that.

But to blatantly block me only because JS execution doesn't verify my humanness? Sketchy at best!

Edit: btw, I'd strip that IP address from that screenshot of yours if I were you, unless it's dynamic and you're due for a new one soon. ;)

At the very least however it allows someone to possibly (roughly) geolocate you.

[pirate]

I've started mocking up what a playwright-based pluginized refactor would look like for ArchiveBox, and I think it's pretty elegant so far! This is still a ways away, but I'm starting to crystalize what I want the plugin-style interface to be between the browser and the extractors.

Please note almost all the classes are stateless namespaces, but I still need to figure out a more elegant composition solution than all this inheritance madness.

from playwright.sync_api import sync_playwright

CRUCIAL_STEPS = (
    BrowserSetupStep,
    PageLoadStep,
)

MINIMAL_STEPS = (
    BrowserSetupStep,
    PageLoadStep,
    TitleRecorderStep,
    HTMLRecorderStep,
    ScreenshotRecordeStep,
)

ALL_STEPS = (
    BrowserSetupStep,
    ExtensionSetupStep,
    SecuritySetupStep,
    ProxySetupStep,

    DialogInterceptorStep,
    TrafficInterceptorStep,
    
    DownloadRecorderStep,
    ConsoleLogRecorderStep,
    WebsocketRecorderStep,
    TrafficRecorderStep,
    TimingRecorderStep,

    PageLoadStep,

    ScriptRunnerStep,
    TitleRecorderStep,
    HTMLRecorderStep,
    PDFRecorderStep,
    TextRecorderStep,
    StorageRecorderStep,
    ScreenshotRecorderStep,
    VideoRecorderStep,
)

r = CompleteRunner()
r.run(url='https://example.com')


class EmptyRunner(BrowserRunner):
    steps = CRUCIAL_STEPS

class MinimalRunner(BrowserRunner):
    steps = MINIMAL_STEPS

class CompleteRunner(BrowserRunner):
    steps = ALL_STEPS


class BrowserRunner:
    steps = ()

    # runtime mutable state
    url = None
    browser = None
    context_args = None
    context = None
    page = None
    config = None

    def run(self, url, config):
        self.url = url
        self.config = config

        self.setup_browser()
        self.setup_context()
        self.setup_page()
        self.run_pre_load()
        self.run_load()
        self.run_post_load()

    def setup_browser(self):
        for step in self.steps:
            step.setup_browser(runner=self)

        return self.browser
        
    def setup_context(self):
        for step in self.steps:
            step.setup_context(runner=self)

    def setup_page(self):
        for step in self.steps:
            step.setup_page(runner=self)

    def pre_load(self):
        for step in self.steps:
            step.pre_load(runner=self)

    def load(self):
        for step in self.steps:
            step.load(runner=self)

    def post_load(self):
        for step in self.steps:
            step.post_load(runner=self)


class BrowserRunnerStep:
    @staticmethod
    def setup_browser(runner):
        pass

    @staticmethod
    def setup_context(runner):
        return {}

    @staticmethod
    def setup_page(runner):
        pass

    @staticmethod
    def run_pre_load(runner):
        pass

    @staticmethod
    def run_load(runner):
        pass

    @staticmethod
    def run_post_load(runner):
        pass


class BrowserSetupStep(BrowserRunnerStep):
    @staticmethod
    def setup_browser(runner):
        runner.browser = sync_playwright.chromium

    @staticmethod
    def setup_page(runner):
        runner.context = runner.browser.launch_persistent_context(**runner.context_args)
        runner.page = runner.context.new_page()

    @staticmethod
    def setup_context(runner):
        runner.context_args = (runner.context_args or {}).update({
            executable_path: "path-to-chromium",
            timeout: 30_000,
        })

class PageLoadStep(BrowserRunnerStep):
    @staticmethod
    def run_load(runner):
        runner.page.goto(url)


class ExtensionSetupStep(BrowserRunnerStep):
    @staticmethod
    def setup_context(runner):
        runner.context_args = (runner.context_args or {}).update({
            args: ["--load-extension: ./my-extension"],
        })


class EmulationSetupStep(BrowserRunnerStep):
    @staticmethod
    def setup_context(runner):
        runner.context_args = (runner.context_args or {}).update({
            headless: True,
            user_agent: runner.config['BROWSER_USER_AGENT'],
            viewport: { 'width': 1280, 'height': 1024 },
            has_touch: False,
            is_mobile: False,
            device_scale_factor: 2,
            locale: 'de-DE',
            timezone_id: 'Europe/Berlin',
            permissions: ['geolocation', 'notifications'],
            geolocation: {"longitude": 48.858455, "latitude": 2.294474},
            color_scheme: 'light',
            **sync_playwright.devices['Pixel 2'],
        })

class SecuritySetupStep(BrowserRunnerStep):
    @staticmethod
    def setup_context(runner):
        runner.context_args = (runner.context_args or {}).update({
            user_agent: 'My user agent',
            java_script_enabled: True,
            chromium_sandbox: True,
            permissions: ['geolocation', 'notifications'],
            extra_http_headers: '...',
            bypass_csp: True,
            ignore_https_errors: True,
        })

class ProxySetupStep(BrowserRunnerStep):
    @staticmethod
    def setup_context(runner):
        runner.context_args = (runner.context_args or {}).update({
            proxy: {
              "server": "http://myproxy.com:3128",
              "username": "usr",
              "password": "pwd",
              "bypass": "github.com,apple.com"
            },
        })


class DialogInterceptorStep(BrowserRunnerStep):
    @staticmethod
    def run_pre_load(runner):
        # handle any dialog boxes
        runner.page.on("dialog", lambda dialog: dialog.accept())

class TrafficInterceptorStep(BrowserRunnerStep):
    @staticmethod
    def run_pre_load(runner):
        # intercept certain requests
        runner.page.route("**/xhr_endpoint", lambda route: route.fulfill(path="mock_data.json"))

class DownloadRecorderStep(BrowserRunnerStep):
    @staticmethod
    def get_context(runner):
        runner.context_args = (runner.context_args or {}).update({
            accept_downloads: True,
            downloads_path: '.',
        })

    @staticmethod
    def run_pre_load(runner):
        # handle any download events
        runner.page.on("download", lambda download: print(download.path()))

class ConsoleLogRecorderStep(BrowserRunnerStep):
    @staticmethod
    def run_pre_load(runner):
        # save console.log to file
        runner.page.on("console", lambda msg: print(msg.text))

class WebsocketRecorderStep(BrowserRunnerStep):
    @staticmethod
    def run_pre_load(runner):
        # handle any websockets opening/closing
        runner.page.on("websocket", lambda websocket: print(
            websocket.url,
            # web_socket.on("close", lambda event: print(event))
            # web_socket.on("framereceived", lambda event: print(event))
            # web_socket.on("framesent", lambda event: print(event))
            # web_socket.on("socketerror", lambda event: print(event))
        ))

class TrafficRecorderStep(BrowserRunnerStep):
    @staticmethod
    def run_pre_load(runner):
        # save requests and responses to file
        runner.page.on("request", lambda request: print(">>", request.method, request.url, request.all_headers()))
        runner.page.on("response", lambda response: print(
            ">>",
            response.request.method,
            response.request.url,
            response.request.headers,
            response.status,
            response.status_text,
            response.url,
            response.headers,
        ))

class TimingRecorderStep(BrowserRunnerStep):
    @staticmethod
    def run_pre_load(runner):
        self.start_time = time.now()

        # measure timing
        runner.page.once("load", lambda: print("page loaded!", self.start_time, time.now()))

class ScriptRunnerStep(BrowserRunnerStep):
    @staticmethod
    def run_post_load(runner):
        # run any scripts in the page
        someresult = runner.page.evaluate('object => object.foo', { 'foo': 'bar' })

        # get page dimensions
        dimensions = runner.page.evaluate('''() => {
          return {
            width: document.documentElement.clientWidth,
            height: document.documentElement.clientHeight,
            deviceScaleFactor: window.devicePixelRatio
          }
        }''')

class TitleRecorderStep(BrowserRunnerStep):
    @staticmethod
    def run_post_load(runner):
        # get title
        return runner.page.title()

class HTMLRecorderStep(BrowserRunnerStep):
    @staticmethod
    def run_post_load(runner):
        # get full page html
        html = runner.page.context()

class TextRecorderStep(BrowserRunnerStep):
    @staticmethod
    def run_post_load(runner):
        # get page innerText
        text = page.inner_text("body")

class StorageRecorderStep(BrowserRunnerStep):
    @staticmethod
    def setup_context(runner):
        runner.context_args = (runner.context_args or {}).update({
            user_data_dir: "/tmp/test-user-data-dir",
            storage_state: "./state.json",
        })

    @staticmethod
    def run_post_load(runner):
        # Save storage state into the file.
        runner.context.storage_state(path="state.json")

class ScreenshotRecorderStep(BrowserRunnerStep):
    @staticmethod
    def run_post_load(runner):
        runner.page.screenshot(path='screenshot.png', full_page=full_page)

class PDFRecorderStep(BrowserRunnerStep):
    @staticmethod
    def run_post_load(runner):
        # generates a pdf with "screen" media type.
        runner.page.emulate_media(media="screen")
        runner.page.pdf(path="page.pdf")

class HARRecorderStep(BrowserRunnerStep):
    @staticmethod
    def setup_context(runner):
        runner.context_args = (runner.context_args or {}).update({
            record_har_omit_content: True,
            record_har_path: './har',
        })

    @staticmethod
    def run_post_load(runner):
        # TODO: save the HAR file path to output dir
        pass


class VideoRecorderStep(BrowserRunnerStep):
    @staticmethod
    def setup_context(runner):
        runner.context_args = (runner.context_args or {}).update({
            record_video_dir: './video',
            slow_mo: 0,
        })

    @staticmethod
    def run_post_load(runner):
        # save the video path
        Path(runner.page.video.path()).move_to('./screenrecording')

[pellaeon]

@pirate Do you have the playwright-based refactor in a public branch? I'd love to contribute if possible. :-)

[pirate]

Not yet but soon! It's just in a gist right now. Will publish it once I've moved >50% of the old codebase into the new structure. I'm traveling in Mexico right now with limited work time but will keep everyone posted as it progresses!

The new design is quite exciting, I'm able to add new features as plugins with <10min of boilerplate work per feature.

https://gist.github.com/pirate/7193ab54557b051aa1e3a83191b69793

[nicoandmee]

@pirate Do you have the playwright-based refactor in a public branch? I'd love to contribute if possible. :-)

Me as well. A few thoughts I've had:

• Network interception disables cache. To reduce network load, you can set up a redis that will store a given request URL as the key, and the response body/headers as the value. Whether that's useful depends on how many things you are archiving I suppose.
• You can load evasions from puppeteer-extra as js to help access pages protected by Cloudfare or another similar service.
• In addition to js-based evasions, ideally, chrome is being launched in headful mode - using Xvfb as a dummy display.
• Chrome and/or puppeteer has a reputation for leaking like a sieve (at least in js land), so it's critical to release resources no longer in use. As an added safeguard, you can also periodically kill and re-spawn the chrome process.

If there's any interest in these things, I can help implement them when the time comes. 👍

[pirate]

Useful scripts for testing and evading archivebox-bot detection blocking with playwright/puppeteer in the future:

• https://github.com/niespodd/browser-fingerprinting#technical-insights-into-bypassing-bot-detection
• https://github.com/matomo-org/device-detector/blob/master/README.md

I'm also leaning towards implementing this using Conifer/Rhizome's well-defined spec for scripted archiving behaviors here: https://github.com/webrecorder/browsertrix-behaviors/blob/main/docs/TUTORIAL.md
Their behaviors framework is also open-source, so if we are compatible with their behaviors our communities can share behavior JS scripts and help grow the capabilities of both tools simultaneously 🎉

[pirate]

Chrome now supports a new framework-agnostic JSON user flow export from the DevTools recording pane. I'd like to use this format if possible instead of playwright/puppeteer directly.

Waiting for a response to see if playwright will implement replay support for it. If so browsertrix-crawler will get support soon after, and I'm likely to just build on top of browsertrix crawler.

Related:

• [Feature] Support replaying Chrome Dev Tools Recorder panel's new JSON session recording format microsoft/playwright#22345
• Support importing behaviors from the new Chrome dev tools Recorder panel JSON export format webrecorder/browsertrix-crawler#283
• https://developer.chrome.com/blog/new-in-devtools-101/#recorder
• https://developer.chrome.com/blog/new-in-devtools-92/#puppeteer-recorder
• https://developer.chrome.com/docs/devtools/recorder/reference/

Otherwise we could also add a custom recorder format for ArchiveBox to the archivebox-extension so that export scripts can be generated directly in our own format (but I prefer the JSON approach above instead^):

• https://developer.chrome.com/blog/extend-recorder/

Part of why this feature is taking so long is that I think all of the solutions for automating JS browser scripting right now are extremely high-maintenance/brittle, and I've been waiting for the ecosystem to mature so as not to overload limited time to work on ArchiveBox by adding a new brittle stack of things I have to maintain.

[pirate]

I've been doing some work for paying ArchiveBox consulting clients to implement advanced puppeteer-based archiving. Here's what I can share so far:

Puppeteer Script to Archive TikTok w/ Auto-login, full-page screenshots, and screenrecording video+audio mp4

My clients (all still non-profits) pay for my time needed to learn about and implement these features, so getting it working for their immediate needs is my priority, but the plan is to integrate these new features back into the main open-source ArchiveBox codebase so everyone can benefit!

[a10kiloham]

For what it's worth this functionality is quite similar to how ChangeDetection works.
https://github.com/dgtlmoon/changedetection.io/tree/master/changedetectionio/content_fetchers
It's very useful to just plug in an additional service into the docker-compose for browserless.io and then pass the websocket URL to the app and get a full browser experience totally headlessly on Ubuntu.