How to Paginate with Mongoose in Node.js [Practical Examples]

Paginate with Mongoose in Node.js

When you have large set of data in your database to be populated on the web page then you will need to restrict the number of entries to be populated on the page. This can be accomplished by making the query to limit the amount of the documents that comes back from MongoDB per request, it is referred as a pagination. There are mainly three ways to implement pagination.

Offset based pagination
Cursor based pagination
Using custom pagination library

Offset based paginate with mongoose

It is a traditional way to paginate with mongoose. Here, It simply uses limit and offset in SQL queries to paginate the data from database. If you are working on NOSQL database, it will be limit and skip. In this approach, client will supply two query parameters in the request: page and limit. Here, Page refers to the current page you are requesting and the Limit is the number of documents to retrieve.

Here, we will first make the connection. Here, we have a simple express server running on port 3000 with a mongoose connection. We will then import a mongoose model Posts so as to use it in the route handler.

require('dotenv').config();
const express = require('express');
const mongoose = require('mongoose');
const Posts = require('./models/posts');

const app = express();
const port = 3000;

mongoose.connect(process.env.MONGO_URI, {
  useNewUrlParser: true,
  useUnifiedTopology: true
});

Now, once the connection is done, we will make a GET request to “/posts”. Here, we are destructing page and limit from the request query. Now, We will need to provide the default values for them if they are not available. It will override otherwise. Here, Limit is the number of documents we want to retrieve. Skip is the amount of documents we want to skip before retrieving our documents. For skip, we will take the page and subtract one, then multiply it by the limit. We will also take the total documents and store it in a variable count so we can calculate total pages by dividing the count by the limit. Finally, we will return the posts, total pages, and current page. As it may cause an exception, we will wrap it in a try/catch block.

app.get('/posts', async (req, res) => {
  // destructure page and limit and set default values
  const { page = 1, limit = 10 } = req.query;

  try {
    // execute query with page and limit values
    const posts = await Posts.find()
      .limit(limit * 1)
      .skip((page - 1) * limit)
      .exec();

    // get total documents in the Posts collection 
    const count = await Posts.count();

    // return response with posts, total pages, and current page
    res.json({
      posts,
      totalPages: Math.ceil(count / limit),
      currentPage: page
    });
  } catch (err) {
    console.error(err.message);
  }
});

There are certain disadvantages to this approach as listed below
It doesn't scale for large datasets. As it scans the record one by one and skip or offset it, If your database has a million records, it will effect scalability. However, if you have real-time data, offset-based pagination will be unreliable and may cause problems of skipping data or data duplication.

Cursor based paginate with mongoose

Cursor-based pagination makes use of a unique record as a cursor for fetching the data. When we pass a cursor and limit, it gets all the data that are less than the cursor value along with the limit.

Here, while implementing the cursor based paginate with mongoose, the cursor value should be sequential or timestamps. So that, we can use comparison operators to fetch the data. Let us now take an example. Here, we will use time as a cursor value which is in descending order. Once user is making request with the limit as 8 for the first time, there will be no cursor value, and it will fetch the most recent value. Hence, DB call will fetch 8+1 value from DB as we need the 9th value as a cursor for the next fetch. Later, we will send the cursor value along with the next request so as to compare that cursor value and fetch data that are less than the cursor value.

const limit = parseInt(req.query.limit)
const cursor = req.query.cursor
let decryptedCursor
let posts

if (cursor) {
  decryptedCursor = decrypt(cursor)
  let decrypedDate = new Date(decryptedCursor * 1000)
  posts = await Trades.find({
    time: {
      $lt: new Date(decrypedDate),
    },
  })
    .sort({ time: -1 })
    .limit(limit + 1)
    .exec()
} else {
  posts = await Trades.find({})
    .sort({ time: -1 })
    .limit(limit + 1)
}
const max = posts.length === limit + 1
let nextCursor = null
if (max) {
  const record = posts[limit]
  var unixTimestamp = Math.floor(record.time.getTime() / 1000)
  nextCursor = encrypt(unixTimestamp.toString())
  posts.pop()
}

res.status(200).send({
  data: tradesCollection,
  paging: {
    max,
    nextCursor,
  },
})

In the above example, we are using max to check if our database has still some data based on the condition. Hence the limit+1 and max will have same value as we are fetching limit+1 data. However, the last value is for the calculation of cursor and should not be sent to the client response.

Using custom pagination library

If you want to have customization in labels with mongoose, you can use mongoose-aggregate-paginate-v2 library. This library provides a page wrapper that helps you alter the return value keys directly in the query itself, thereby eliminating the efforts for code transformation.

In order to use this, you will need to add plugin to a schema and then use model paginate method. You can also use custom return labels as per requirement of your application. However, you will require to specify the this changes in field names using customLabels object in options.

const mongoose = require('mongoose');
const mongoosePaginate = require('mongoose-paginate-v2');
const postschema= new mongoose.Schema({name:String});

postschema.plugin(mongoosePaginate);

const model = <
  postdoc,
  PaginateModel
>('Posts', postschema, 'posts');

const posts = {
	page: 1,
	limit: 20,
};

model.paginate({}, options, function(err,result){});

Summary

The knowledge of paginate with mongoose in nodejs is very useful while working on real time applications. In many situations, when the number of records are more and cannot fit on the page, we will need to paginate to give decent look to the page. In this tutorial, we covered three different approach to do paginate with Mongoose in node js. As per the requirement of an application, we can choose an appropriate approach for pagination. We learned in detail about this with an example. All in all, this tutorial, covers everything that you need to know in order to have a clear view on how to paginate with mongoose in nodejs.

References

Mongoosejs Docs
Mongoose Paginate V2