Node.js loop through files in directory [SOLVED]

Different methods in Node.js to loop through files in directory

Node.js, with its built-in filesystem module 'fs', is a powerful tool to manage files and directories. One common operation in server-side scripting or automation tasks is looping through files in a directory. It's often crucial to read, manipulate, or analyze the files in a directory and Node.js makes this process efficient and streamlined.

In this article, we are going to cover different methods to loop through files in a directory using Node.js. Although there are multiple ways to achieve this, the methods often vary based on specific use cases, underlying system architecture, and individual project requirements. Each method comes with its own strengths and constraints, and understanding these differences can help in selecting the most suitable method for a specific task.

1. fs.readdir() method: This is the simplest and most straightforward method provided by Node.js's fs module to read the contents of a directory. The method returns an array of filenames of all files in the directory. However, it doesn't support recursive reading of subdirectories.
2. fs.readdirSync() method: This method is the synchronous version of fs.readdir(). It blocks the execution of subsequent code until it finishes reading the directory, making it a less-preferred choice for large directories where non-blocking operation is critical.
3. fs.promises.readdir() method: With the advent of Promises in Node.js, the fs module provides a promises API that allows the use of async/await syntax, making the code more readable and easier to manage, especially for larger directories and deeper nested callbacks.
4. fs.opendirSync() method: A synchronous method that opens a directory and allows reading its contents with the returned fs.Dir object.
5. Recursive Directory Reading: By default, the readdir() method doesn't read files in subdirectories. If we need to recursively read files in subdirectories, we will need to implement a custom recursive function or use third-party packages like 'recursive-readdir' or 'kLaw'.
6. Stream-based Reading with fs.createReadStream(): When working with large files or directories, it's often better to read data in chunks to prevent overloading the system's memory. This method uses streams to read data chunk by chunk.
7. Using Glob Patterns with 'glob' package: Sometimes, we want to read files based on specific patterns or extensions. The 'glob' module allows us to do this by providing a way to read files in a directory based on glob patterns.

 

 

1. Using fs module

The fs module enables reading or writing to the filesystem.

The built-in module has multiple functions to simplify working with the file system. You can view all the methods and properties by entering the REPL mode and typing fs followed by a dot . and a double-press on the Tab key.

 

1.1 Using fs.readdir() method

The fs.readdir() method asynchronously reads the contents of a directory. The method returns an array of filenames of all files in the directory.

const fs = require('fs');

fs.readdir('./folder', (err, files) => {
  if (err) throw err;
  console.log(files);
});

Next let's iterate through each file inside the directory and read the content:

const fs = require('fs');
const path = require('path');

fs.readdir('/tmp/test', (err, files) => {
  if (err) throw err;

  files.forEach(file => {
    const filePath = path.join('/tmp/test', file);
    fs.readFile(filePath, 'utf8', (err, contents) => {
      if (err) throw err;
      console.log(`Content of ${file}: ${contents}`);
    });
  });
});

Output:

Content of 1.txt: hello

Content of 2.txt: hello2

Content of 3.txt: hello3

Content of 4.txt: hello4

 

1.2 fs.readdirSync() method

This is the synchronous version of fs.readdir(). It blocks the execution of subsequent code until it finishes reading the directory.

const fs = require('fs');

try {
  const files = fs.readdirSync('./folder');
  console.log(files);
} catch (err) {
  console.error(err);
}

Next let's loop through each file inside the directory and read the content of each file:

const fs = require('fs');
const path = require('path');

try {
  const files = fs.readdirSync('/tmp/test');
  files.forEach(file => {
    const filePath = path.join('/tmp/test', file);
    const contents = fs.readFileSync(filePath, 'utf8');
    console.log(`Content of ${file}: ${contents}`);
  });
} catch (err) {
  console.error(err);
}

 

1.3 fs.promises.readdir() method

The fs.promises.readdir() method allows you to use async/await syntax. It's particularly useful when dealing with larger directories and deeper nested callbacks.

const fs = require('fs').promises;

async function printFiles() {
  try {
    const files = await fs.readdir('./folder');
    console.log(files);
  } catch (err) {
    console.error(err);
  }
}

printFiles();

Let's loop through each file and read the content:

const fs = require('fs').promises;
const path = require('path');

async function printFiles() {
  try {
    const files = await fs.readdir('/tmp/test');
    for (let file of files) {
      const filePath = path.join('/tmp/test', file);
      const contents = await fs.readFile(filePath, 'utf8');
      console.log(`Content of ${file}: ${contents}`);
    }
  } catch (err) {
    console.error(err);
  }
}

printFiles();

 

1.4 fs.opendirSync() method

opendirSync() is another method provided by the Node.js fs module that you can use to synchronously open a directory. The opendirSync() method returns an fs.Dir object, from which you can call the readSync() method to read the contents of the directory.

const fs = require('fs');

const dir = fs.opendirSync('./directoryPath');
let dirent;

while ((dirent = dir.readSync()) !== null) {
  console.log(dirent.name);
}

dir.closeSync();

In this code:

• fs.opendirSync() opens the directory synchronously and returns an fs.Dir object.
• dir.readSync() reads the next directory entry synchronously. When there are no more directory entries, dir.readSync() returns null, which ends the loop.
• dirent.name provides the name of the directory entry.
• dir.closeSync() is called after the loop to close the directory.

To read the content of individual file inside the directory:

const fs = require('fs');
const path = require('path');

const dir = fs.opendirSync('/tmp/test');
let dirent;

while ((dirent = dir.readSync()) !== null) {
  if (!dirent.isDirectory()) {
    const filePath = path.join('/tmp/test', dirent.name);
    const contents = fs.readFileSync(filePath, 'utf8');
    console.log(`Content of ${dirent.name}: ${contents}`);
  }
}

dir.closeSync();

 

2. Recursive Directory Reading

The readdir() method doesn't read files in subdirectories by default. To do so, you need to implement a custom recursive function.

const fs = require('fs');
const path = require('path');

function readDirRecursive(dirPath) {
  fs.readdir(dirPath, (err, files) => {
    if (err) throw err;

    files.forEach(file => {
      const filePath = path.join(dirPath, file);
      fs.stat(filePath, (err, stats) => {
        if (err) throw err;

        if (stats.isDirectory()) {
          readDirRecursive(filePath);
        } else {
          console.log(filePath);
        }
      });
    });
  });
}

readDirRecursive('./folder');

We can enhance the logic to read the content of each file:

const fs = require('fs');
const path = require('path');

function readDirRecursive(dirPath) {
  fs.readdir(dirPath, (err, files) => {
    if (err) throw err;

    files.forEach(file => {
      const filePath = path.join(dirPath, file);
      fs.stat(filePath, (err, stats) => {
        if (err) throw err;

        if (stats.isDirectory()) {
          readDirRecursive(filePath);
        } else {
          fs.readFile(filePath, 'utf8', (err, contents) => {
            if (err) throw err;
            console.log(`Content of ${file}: ${contents}`);
          });
        }
      });
    });
  });
}

readDirRecursive('/tmp/test');

 

3. Using Glob Patterns with 'glob' package

The 'glob' module is not a built-in Node.js module; it's a third-party module that you need to install using a package manager like npm (Node Package Manager) or Yarn.

You can install it by running the following command in your project's root directory:

npm install glob

OR

yarn add glob

After running this command, Node.js should be able to find and require the 'glob' module without any issues. The 'glob' module allows you to read files in a directory based on glob patterns.

const glob = require('glob');

glob('./folder/**/*.txt', (err, files) => {
  if (err) throw err;
  console.log(files);
});

To loop through each file and read it's content we can use forEach loop:

const glob = require('glob');
const fs = require('fs');
const path = require('path');

glob('/tmp/test/**/*.txt', (err, files) => {
  if (err) throw err;

  files.forEach(file => {
    fs.readFile(file, 'utf8', (err, contents) => {
      if (err) throw err;
      console.log(`Content of ${path.basename(file)}: ${contents}`);
    });
  });
});

 

Summary

I could find alot of questions posted on stackoverflow.com wherein users wanted to understand how to loop through files in directory in Node.js and JavaScript.

Looping through files in a folder Node.JS
Iterate over files in a folder (javascript)
how can I loop through files in a folder and subfolders using electron?

So I decided to have this tutorial. A Node.js loop through files in directory is possible through the while, for, and for-await-of loop with the fs module's methods like opendirSync(), readdir(), and fs.promises.opendir().