Node.js get all files in directory recursively [SOLVED]

The simplest way to achieve a Node.js get all files in directory and subdirectories recursively is through the glob module.

glob([directory] + [search-pattern], [options], [callback function])

Where;

• directory: path to the target files and subdirectories,
• search pattern: shell-like regular expressions,
• options: controls the depth of search, and
• callback function: returns either an error object or an array of matching files.

Here is more you should know about the glob module, its practical examples, and alter ways to Node.js get all files in directory and subdirectories.

 

How the glob module works

The glob module enables searching a directory's contents using special patterns. Here are some of the patterns.

• *: match nil or multiple characters in the given file path.
• *(a|b|c): match nil or multiple occurrences of the specified pattern.
• ?: match one character.
• @(pattern|pat*|pat?erN): match one of the specified patterns.
• +(pattern|pattern|pattern): match one or multiple occurrences of the specified patterns.
• [...]: match a range of characters, except when the range's first character is ! or ^. In that case, the algorithm matches other characters outside the range.

The glob module returns an array of (file or subdirectory) matches or an empty array [] if it finds no match.

 

Lab environment setup

Let's set up a lab environment to Node.js get all files in directory and subdirectories. You can run the following commands on Mac, Linux, or (Windows) Git Bash terminal.

Create the project directory and the main script file

$ mkdir getAllFiles && cd getAllFiles
$ touch index.js

Make a subdirectory housing a typical frontend environment with static files in public and an images directory. The public folder contains index.html, style.css, and script.js.

$ mkdir subdirectory && cd subdirectory
$ mkdir public images
$ cd public
$ touch index.html style.css script.js

Now return to the project directory and open it with Visual Studio Code.

$ cd ../..
$ code .

In the following sections, we get the files and subdirectories in the subdirectory after running the index.js file with the node command.

node index.js

 

Examples of Node.js get all files in directory and subdirectories using the glob module

Before using the module

Open Visual Studio Code's integrated terminal, then initialize an NPM package before installing the glob module.

Input

npm init -y
npm i glob

Output

user@hostname:~/getAllFiles$ npm init -y
Wrote to /home/user/getAllFiles/package.json:

{
  "name": "getallfiles",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}


user@hostname:~/getAllFiles$ npm i glob

added 9 packages, and audited 10 packages in 8s

1 package is looking for funding
  run `npm fund` for details

found 0 vulnerabilities

 

After using the glob module

Let's use the glob module in the index.js file to get files and subdirectories using various search patterns.

Example~1: Node.js get recursively all files

Input

const glob = require('glob')

const targetDir = './subdirectory'

glob(targetDir + '/**/*', (err, files) => {
  if (err) {
    console.log('Error', err)
  } else {
    console.log(files)
  }
})

Output

user@hostname:~/getAllFiles$ node index.js 
[
  './subdirectory/images',
  './subdirectory/public',
  './subdirectory/public/index.html',
  './subdirectory/public/script.js',
  './subdirectory/public/style.css'
]

We get an array of all the files

'./subdirectory/public/index.html',
'./subdirectory/public/script.js',
'./subdirectory/public/style.css'

and subdirectories

'./subdirectory/images',
'./subdirectory/public',

of the main subdirectory.

 

Example~2: Node.js get recursively all files including the dot files

Let's add a .dockerignore file in the subdirectory and read the subdirectory's contents, including the dot file.

touch subdirectory/.dockerignore

Input

const glob = require('glob')

const targetDir = './subdirectory'

glob(targetDir + '/**/*', {dot: true}, (err, files) => {
  if (err) {
    console.log('Error', err)
  } else {
    console.log(files)
  }
})

We add the dot:true option before the callback function to enable the reading of the dot files.

Output

user@hostname:~/getAllFiles$ node index.js 
[
  './subdirectory/.dockerignore',
  './subdirectory/images',
  './subdirectory/public',
  './subdirectory/public/index.html',
  './subdirectory/public/script.js',
  './subdirectory/public/style.css'
]

The system won't return the .dockerignore file without the dot: true option.

 

Example~3: Node.js get all files in the public directory with specific extension

Assume we want to generate an array of .js files only from the public directory. Let's introduce a fetch.js file in the directory.

touch subdirectory/public/fetch.js

Then update the target directory to be the public directory.

const targetDir = './subdirectory/public'

And the search pattern to include the .js extension.

Input

const glob = require('glob')

const targetDir = './subdirectory/public'

glob(targetDir + '/**/*.js', {dot: true}, (err, files) => {
  if (err) {
    console.log('Error', err)
  } else {
    console.log(files)
  }
})

Output

user@hostname:~/getAllFiles$ node index.js 
[ './subdirectory/public/fetch.js', './subdirectory/public/script.js' ]

We get an array of files ending in the .js extension.

The above examples demonstrate the typical ways to Node.js get all files in directory and subdirectories. What if we want to get all files and subdirectories without importing any external module?

Yes, we can do that, as shown below.

 

Bonus tricks

Although the glob module simplifies getting files and subdirectories in Node.js directory, we could write a custom code to achieve the same result without installing a third-party package.

Here is an example.

Input

const { readdirSync, statSync } = require('fs')
const { join } = require('path')

const allFilesAndSubdirectories = []

const readTargetDir = directory => {

  readdirSync(directory).forEach( file => {

      const absoluteFilepath = join(directory, file)

      if (statSync(absoluteFilepath).isDirectory()) {
        return readTargetDir (absoluteFilepath)
      } else {
        return allFilesAndSubdirectories.push(absoluteFilepath)
      }

  })

}

readTargetDir('./subdirectory/')

console.log(allFilesAndSubdirectories)

 

We import the fs module's readdirSync() method and statSync property and the path module's join() method.

const { readdirSync, statSync } = require('fs')
const { join } = require('path')

Using the readdirSync() method, we read the contents of the specified directory.

  readdirSync(directory).forEach( file => {

      const absoluteFilepath = join(directory, file)

      if (statSync(absoluteFilepath).isDirectory()) {
        return readTargetDir (absoluteFilepath)
      } else {
        return allFilesAndSubdirectories.push(absoluteFilepath)
      }

  })

We then loop through the returned array and check whether the array element is a directory. If so, we recursively loop through the element before pushing its contents into the allFilesAndSubdirectories array.

If the array element is not a directory, we don't recursively loop its contents. Instead, we push the filename into the allFilesAndSubdirectories array. The absoluteFilepath joins each of the array elements to the main subdirectory subdirectory.

Finally, we call the readTargetDir() function with the target directory and console-log the contents of the allFilesAndSubdirectories array.

Output

user@hostname:~/getAllFiles$ node index.js 
[
  'subdirectory/.dockerignore',
  'subdirectory/public/fetch.js',
  'subdirectory/public/index.html',
  'subdirectory/public/script.js',
  'subdirectory/public/style.css'
]

We get all files and subdirectories of the main subdirectory.

 

Key takeaway

The glob module simplifies a Node.js get all files in directory and subdirectories. Alternatively, you can build a custom module to get all files and subdirectories after understanding the fs and path modules.

 

References

Get all files recursively in directories NodejS
Node.js fs.readdir recursive directory search
nodejs read directory recursively, get files paths, then use paths to copy files to another directory