upload media files to strapi within a node process

I enjoy the ease of use strapi.io. At @boldventures we started in may 2020 with our first strapi project and today (august 2020) the count of strapi projects is on three with more to come.

On my band website I am switchting from my nuxt.js/express.js-api to nuxt.js/nuxtjs-strapi-module/strapi jamstack. Moving the in-template content to strapi and the media handling too. Secured behind a login, all band members have access to a rehearsal archive (starting from 2014) and (soon) to all songs we play with lyrics, chords, notes, etc.

the challenge

I record every rehearsal, post processing with audition (normalize, equalize, multiband compress) and enrich meta-data with kid3. After that I push all mp3s to my server and let a node script import all new songs.

Now I want to store all mp3s directly in strapi and let my node script push all data (metadata and binary data) through strapis restful api.

In this post I demonstrate the process on a simple example data, not my full band usecase. Starting with public/unauthenticated data upload and later with authenticated upload. A working code example is available on github: strapi-node-upload.

strapi setup

npx create-strapi-app upload-test is the starting point for this journey. I used the --quickstart option to get me going fast. For production make sure a mysql or mongo db is in use.

Create an admin user jump directly in to the content-types builder section and create a new collection type called song:

In this simple example a song will have a short text field title and a single media file  mp3.

Now I can handle mp3 files via strapis backend. But the importer will use the same structure and will do the most work for me.

To use any strapi restful endpoint, it is needed to allow the use of this collection types endpoint. In the first step I allow unauthenticated users to update and create data vie the endpoint. Later I will change this to allow only authenticated users to update and create song entries.

node importer setup

In my usecase I use the music-metadata package to read alle mp3 data and store them in my database, but in this example I use the filename as title and the file itself. I hope this way, this example is mor versatile for your usecase. I use node v12.17.0 for this code.

I create a file called node-import.js which will do the following:
*  reads all files in directory ./import-files
* uploads each file as song entry to strapi using node

node-import.js

We need the the following packages:
* strapi file upload documentation
* node/fs package - for file system access
* form-data package - to prepare data we send to strapi npm install form-data --save
* node/http|https - for post request
* dotenv package - for .env file handling

// @see https://github.com/djpogo/strapi-node-upload
// node-import.js
// require fs for filesystem access
const fs = require('fs');

// create form-data body for strapi
const FormData = require('form-data');

// for Non-SSL/localhost strapi endpoint we use http
const http = require('http');

// for SSL/production strapi endpoint we use https
const https = require('https');

const importDir = './import-files';
const strapiUrl = 'http://localhost:1337/';

const httpClient = strapiUrl.indexOf('https') === 0 ? https : http;

The import functionality looks like this:

// @see https://github.com/djpogo/strapi-node-upload
// node-import.js continued

// read directory contents
fs.readdir(importDir, (error, files) => {
  // handle errors
  if (error) {
    console.log('Error!', error);
  } else {
    // walk through all files
    files.forEach((file) => {
      // on production better check value of `file` to be something that should be processed
      
      // prepare data to send to server
      const form = new FormData();
      // watch this `data` property here. It contains all of non binary data of your object and it must be JSON.stringified()!
      form.append('data', JSON.stringify({
        title: file
      }));
      
      // @see https://strapi.io/documentation/v3.x/plugins/upload.html#upload-file-during-entry-creation
      // the binary data must be prefixed with files.<strapi-field-name> and be a readableStream object
      form.append('files.mp3', fs.createReadStream(`${importDir}/${file}`), file);

      const request = httpClient.request(
        `${strapiUrl}`,
        {
          method: 'post',
          path: '/songs', // this is your collection entity name
          headers: form.getHeaders(),
        }, (res) => {
          res.on('error', (error) => {
            console.log(`Error on file ${file}`, error);
          });
        });
        // send data to server
        form.pipe(request);
    });
  }
});

All starts with an async directory reading of import-files folder fs.readdir(<path>, function callback(error, files)...). This function returns an Array with file names or an error object.
Than it walks through every file files.forEach() where it should check that the entry is not a directory or a file you do not want to process.
For every file it creates a formData Object, containing a data part for all non binary data and a files.xyz part for that file. Ensure to JSON.stringify(...) the data formData part and that the file field is prefixed with files. in my example files.mp3, because my file field is called mp3 in strapi.
When strapi is running npm run develop  and this script is executed node node-import.js entries are created.

To restrict this import workflow to a specific user group a litte more needs to be done and not much more than a little!

secure the api

Before opening all doors and let everyone create, update or delete our data, let's restrict access to these actions.
First create a new role for our api users (machines):

The new role is allowed to do everything with the song entity. Now remove these actions from the public user group!

After removing these permissions, a run of the import script will produce no new entries in strapi. The api is not open anymore.

Now create a new api user belonging to the api user group:

getting api users jwt

Next thing is to get an access token for that user. The login docu gives all we need to get the api users jwt. I use Insomnia to make the authentication post request to get to json web token (jwt):

Update: I chatted with my colleagues about this jwt handling and the better way is to authenticate the api user within the script, to get always a working jwt, instead of updating the .env file every month.
I updated the code on github accordingly to make the authentication call.
Soon I will add an axios example for this workflow too, because in my usecase I will use axios. But for this example I want to be as basic as possible. Maybe I will get even rid of the form-data in the pure node code.

Another idea is to provide the credentials to the import script, so it can authorises itself and get everytime a fresh jwt.
Both ways - if a user account needs to be teared down, there is a blocked toggle for every user in the backend.

The jwt needs to be added as an authorization header into the import code and our importer will run again:

// .env
JWT=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MSwiaWF0IjoxNTk2Mzc5MzkwLCJleHAiOjE1OTg5NzEzOTB9.YSf_PLjzwYrWs4DHteUFbs6P6GGdk_xy7Kacg-5GkBc

// node-import.js
...

const jwt = process.env.JWT; 

...
    const request = httpClient.request(
      `${strapiUrl}`,
      {
        method: 'post',
        path: '/songs',
        headers: {
            ...form.getHeaders(),
            Authorization: `Bearer ${jwt}`,
        },
      }, (res) => {
          res.on('error', (error) => {
            console.log(`Error on file ${file}`, error);
          });
        });
        form.pipe(request);
    });

Thanks for reading. A full working code example is available here: https://github.com/djpogo/strapi-node-upload.

Article Image from Felix Mittermeier via unsplash and ghost ♥.