Ufdjrw

I have a long txt file with ~1000 urls, which needs to be executed, to warm the varnish cache.

Since I need puppeteer, is that there is important content loaded by AJAX call.

This is my first attempt, but not master in node.

The real issue is that it make a 100% load, and starts too many threads.

const puppeteer = require('puppeteer');
const readline = require('readline');
const fs = require('fs');

const rl = readline.createInterface(
 input: fs.createReadStream('varnish-warmer.txt')
 );

rl.on('line', (line) => 
(async () => 
 if (line != '') 
 const browser = await puppeteer.launch();
 const page = await browser.newPage();
 await page.goto(line);
 await page.waitFor(1000);

 browser.close();
 

)();
);

As you already noticed, your code launches all browsers in parallel which overloads your system. You could either visit each URL one after another (option 1) or use a pool of browsers to speed the process up (option 2).

Option 1

Launches one browser and visits all pages one after another:

const puppeteer = require('puppeteer');
const fs = require('fs');

const lines = fs.readFileSync('varnish-warmer.txt').toString().split('n');

(async () => 
 const browser = await puppeteer.launch();
 const page = await browser.newPage();
 for (const line of lines) 
 await page.goto(line);
 await page.waitFor(1000);
 
 await browser.close();
)();

Option 2

As option 1 might take a while for 1000 URLs, you might want to use a pool of browsers to visit the pages in parallel and speed things up. You can use puppeteer-cluster for that (disclaimer: I'm the author of the library).

const Cluster = require('puppeteer-cluster');
const fs = require('fs');

(async () => 
 const cluster = await Cluster.launch(
 concurrency: Cluster.CONCURRENCY_BROWSER,
 maxConcurrency: 10, // how many URLs should be visited in parallel
 // monitor: true, // uncomment to see information about progress
 );

 // Define the task for each URL
 await cluster.task(async ( page, data: url ) => 
 await page.goto(url);
 await page.waitFor(1000);
 );

 // Queue the URLs
 const lines = fs.readFileSync('varnish-warmer.txt').toString().split('n');
 lines.forEach(line => cluster.queue(line));

 // Wait for the tasks to finish and close the cluster after that
 await cluster.idle();
 await cluster.close();
)();

You can play around with the value of maxConcurrency to change the number of workers depending on the capabilities (CPU/memory) of your system.