Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Question) Tasks separated into modules for each website #540

Open
4e576rt8uh9ij9okp opened this issue May 3, 2024 · 0 comments
Open

(Question) Tasks separated into modules for each website #540

4e576rt8uh9ij9okp opened this issue May 3, 2024 · 0 comments

Comments

@4e576rt8uh9ij9okp
Copy link

Hi there,
I'm new to web-crawling and I would like to use puppeteer-cluster and separate tasks into it's own nodejs modules to keep the tasks separate from each other.

/index.js
/tasks/google.js
/tasks/youtube.js

index.js

const { Cluster } = require('puppeteer-cluster');
const googleTask = require('./tasks/google.js')
const youtubeTask = require('./tasks/youtube.js')

(async () => {
    // Create a cluster with 2 workers
    const cluster = await Cluster.launch({
        concurrency: Cluster.CONCURRENCY_CONTEXT,
        maxConcurrency: 2,
    });

    // Define a task (in this case: screenshot of page)
    await cluster.task(googleTask);
    await cluster.task(youtubeTask);

    // Shutdown after everything is done
    await cluster.idle();
    await cluster.close();
})();

./tasks/google.js

module.exports = async () => {
        await page.goto("https://google.com");
        // do something
    }
@4e576rt8uh9ij9okp 4e576rt8uh9ij9okp changed the title (Question) Task separation for each website (Question) Task separated into modules for each website May 3, 2024
@4e576rt8uh9ij9okp 4e576rt8uh9ij9okp changed the title (Question) Task separated into modules for each website (Question) Tasks separated into modules for each website May 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant