How to display Markdown files in nested folders in Next.JS?

Question

In my Next.JS project, I have many Markdown files distributed in many folders which are considered as categories. So I have folders called 'CategoryOne' and 'CategoryTwo' at the root of the project (at the same level as node_modules, pages, public). Each of these folders contains some Markdown files.

I have a page called articles which should render all .md files from 'CategoryOne' and 'CategoryTwo'. As I have to get them, articles is a folder which contains index.js (lists all .md files as articles) and [slug].js (displays the article you clicked).

Let's take a look! :

pages/articles/index.js

import React, { useState } from 'react'
import fs from 'fs'
import path from 'path'
import matter from 'gray-matter'
import { useRouter } from 'next/router'

export default function Blog({ posts, postsTwo }) {
    const router = useRouter()
    if (router.isFallback) {
        return <div>Loading...</div>
    }
    return (
        <>
            {/* Maps articles from both Categories */}
        </>
    )
}

export async function getStaticProps() {
    const files = fs.readdirSync(path.join('CategoryOne'))
    const posts = files.map((filename) => {
        const slug = filename.replace('.md', '')
        const markdownWithMeta = fs.readFileSync(
            path.join('CategoryOne', filename),
            'utf-8'
        )
        const { data: frontmatter } = matter(markdownWithMeta)

        return {
            slug,
            frontmatter,
        }
    })

    const files2 = fs.readdirSync(path.join('CategoryTwo'))
    const posts2 = files2.map((filename) => {
        const slug2 = filename.replace('.md', '')
        const markdownWithMeta2 = fs.readFileSync(
            path.join('CategoryTwo', filename),
            'utf-8'
        )
        const { data: frontmatter } = matter(markdownWithMeta2)

        return {
            slug2,
            frontmatter,
        }
    })

    return {
        props: {
            posts: posts,
            postsTwo: posts2,
        },
    }
}

[slug].js

import React, { useState, useEffect } from 'react'
import fs from 'fs'
import path from 'path'
import matter from 'gray-matter'
import marked from 'marked'
import { useRouter } from 'next/router'

export default function PostPage({
    frontmatter: { title, date, reading_time, category },
    slug,
    content,
    posts,
}) {
    
    if (router.isFallback) {
        return <div>Loading...</div>
    }

    return (
        <>
            <div>
                <h1>{title}</h1>
                <p>{date} • {reading_time}</p>
                <div>
                    <div dangerouslySetInnerHTML={{__html: marked(content)}}></div>
                </div>
           </div>
        </>
    )
}

export async function getStaticPaths() {
    const files = fs.readdirSync(path.join('CategoryOne'))
    const paths = files.map((filename) => ({
        params: {
            slug: filename.replace('.md', ''),
        },
    }))

    return {
        paths,
        fallback: false,
    }
}

export async function getStaticProps({ params: { slug } }) {
    const files = fs.readdirSync(path.join('CategoryOne'))
    const markdownWithMeta = fs.readFileSync(
        path.join('CategoryOne', slug + '.md'),
        'utf-8'
    )
    const { data: frontmatter, content } = matter(markdownWithMeta)
    const posts = files.map((filename) => {
        const slug = filename.replace('.md', '')
        const markdownWithMeta = fs.readFileSync(
            path.join('CategoryOne', filename),
            'utf-8'
        )
        const { data: frontmatter } = matter(markdownWithMeta)

        return {
            slug,
            frontmatter,
        }
    })
    return {
        props: {
            frontmatter,
            slug,
            content,
            posts,
        },
    }
}

When I click on an article from 'CategoryTwo' I got a 404 error. It's because in [slug].js I only could return slug from 'CategoryOne'..

In index.js, getStaticProps() is too long, I would like to get this function recursive. Because I think it would be better if I put the Categories folders into one folder called 'posts'. So if there are more categories, I could get all .md files whitout calling them by their names.

/posts
  |______ /CategoryOne
  |______ /CategoryTwo

To sum up, we have to focus on the getStaticProps(), getStaticPaths() and filesystem libray to recursively list files from subdirectories.

[EDIT • September 10th 2021]

I made two functions into my utils/index.js. First one get the subfolders, and the second one dive into each subfolder.

import { readdir } from 'fs/promises';
const postsDirectory = path.join(process.cwd(), 'posts');

export function getDirectories(postsDirectory) {
  return fs
    .readdirSync(postsDirectory, { withFileTypes: true })
    .filter((dirent) => dirent.isDirectory())
    .map((dirent) => dirent.name);
}

export async function getAllFiles() {
  const test = getDirectories(postsDirectory);
  let table = [];
  for (const folder of test) {
    const subfolder = path.join(postsDirectory, folder);
    let files = await readdir(subfolder);
    for (const file of files) {
      table.push({ file: file, folder: folder });
    }
  }
  return table;
}

/* Result

[
  { file: 'markdown-one-1.md', folder: 'CategoryOne' },
  { file: 'markdown-one-2.md', folder: 'CategoryOne' },
  { file: 'markdown-one-1.md', folder: 'CategoryTwo' },
  { file: 'markdown-one-2.md', folder: 'CategoryTwo' },
]

*/

I found that if you have some subfolders inside your posts folder, you should have the same in your posts pages as the schema below:

├── node_modules
├── package.json
├── pages
│   ├── index.js
│   ├── _app.js
│   └── posts
│       ├── CategoryOne
│           ├── [id].js //to display .md files of CategoryOne
│       ├── CategoryTwo
│           ├── [id].js //to display .md files of CategoryTwo
├── ..
└── posts
    ├── CategoryOne
    │   ├── markdown-one-1.md
    │   ├── markdown-one-2.md
    │   ├── markdown-one-3.md
    │   └── ..
    └── CategoryTwo
        ├── markdown-two-1.md
        ├── markdown-two-2.md
        ├── markdown-two-3.md
        └── ..

There should be a way to automate the creation of those [id].js files depending on how much subfolders you have..

Answer 1

Like you said, it would be better to put your categories and their posts in their own directory. I know you mentioned doing this recursively, but if that is not an absolute requirement, I would suggest using glob.

Assuming you had the below file structure 👇

├── node_modules
├── package.json
├── ..
└── posts
    ├── CategoryOne
    │   ├── markdown-one-1.md
    │   ├── markdown-one-2.md
    │   ├── markdown-one-3.md
    │   └── ..
    └── CategoryTwo
        ├── markdown-two-1.md
        ├── markdown-two-2.md
        ├── markdown-two-3.md
        └── ..

you could then do something like this in your getStaticProps and/or getStaticPaths methods:

const glob = require('glob');

glob('posts/**/*.md', (err, files) => {
    console.log({files});

    /*
     * will produce:
     * {
     *     files: [
     *       'posts/CategoryOne/markdown-one-1.md',
     *       'posts/CategoryOne/markdown-one-2.md',
     *       'posts/CategoryOne/markdown-one-3.md',
     *       'posts/CategoryTwo/markdown-two-1.md',
     *       'posts/CategoryTwo/markdown-two-2.md',
     *       'posts/CategoryTwo/markdown-two-3.md'
     *     ]
     * }
     */

});

You would then just need to split('/') on each of those files, organize the category/file names and return them as props.