multer-remote-storage
TypeScript icon, indicating that this package has built-in type declarations

0.0.6 • Public • Published

multer-remote-storage

Use Multer to easily upload files to Cloudinary, AWS S3, or Google Cloud Storage

  • Easily switch between storage targets
  • Use your own validator function to prevent file uploads in the event other HTTP request body data does not pass validation
  • Add a chunk_size to the options to automatically trigger a chunked upload for your file

Contents

  1. Setup
  2. Examples
  3. Options
  4. Validation
  5. Public Id
  6. TypeScript Example

Setup

Run this command to install:

npm install multer-remote-storage

Examples

Cloudinary Example

storage.js

import {v2 as Cloudinary} from 'cloudinary';
import { RemoteStorage } from 'multer-remote-storage';

Cloudinary.config({
    cloud_name:process.env.CLOUDINARY_NAME,
    api_key:process.env.CLOUDINARY_API_KEY,
    api_secret:process.env.CLOUDINARY_SECRET,
    secure:true,
});

const storage = new RemoteStorage({
    client:Cloudinary,
    params: {
        folder:'Myfolder'
    }
});

export { storage }

route.js

import multer from 'multer';
import { storage } from '../utilities/storage.js';
import {filter} from '../utilities/validators/fileValidator.js'; //Multer-related
import { topicValidation } from '../utilities/validators/middleware/validators.js'; //JOI

const router = express.Router({ caseSensitive: false, strict: false });

const parser = multer({storage, fileFilter:filter, limits:{fileSize:1024000}});

router.route('/:username/create')
  .post(isLoggedIn,isAuthor,parser.single('topic[file]'),topicValidation,createTopic)
Property Value
client Cloudinary namespace
params? Same as Cloudinary's upload_stream and upload_chunked_stream
options? See Below

The following data will be appended to req.file

variable data type info
etag string
filename string includes folder, excludes extension
folder string
height number if applicable
width number if applicable
path string public URL
signature string
size number
timeCreated string
versionId string

Google Cloud Storage Example

storage.js

import {join,dirname} from 'path';
import { Storage as Gcs } from '@google-cloud/storage';
import { RemoteStorage } from 'multer-remote-storage';

const __filename = fileURLToPath(import.meta.url);

const gcsStorage = new RemoteStorage({
    client:new Gcs({
        keyFilename: join(dirname(__filename),'pathToKey.json'),
        projectId: 'your-google-storage-projectId',
    }),
    params:{
        bucket:'mybucket'
    }
});

export { gcsStorage }

router.js

import multer from 'multer';
import { gcsStorage } from '../utilities/storage.js';
import {filter} from '../utilities/validators/fileValidator.js'; //Multer-related
import { topicValidation } from '../utilities/validators/middleware/validators.js'; //JOI

const router = express.Router({ caseSensitive: false, strict: false });

const gcsParser = multer({storage:gcsStorage, fileFilter:filter, limits:{fileSize:1024000}});

router.route('/:username/create')
  .post(isLoggedIn,isAuthor,gcsParser.single('topic[file]'),topicValidation,createTopic)
Property Value
client Google Cloud's Storage Class
params bucket (required) PLUS options for uploading object to bucket
options? See Below

The following data will be appended to req.file

variable data type info
bucket string
contentType string
etag string
filename string
path string public URL
size number
storageClass string
timeCreated string

AWS S3 Example

storage.js

import { S3Client } from '@aws-sdk/client-s3';
import { RemoteStorage } from 'multer-remote-storage';

const s3Storage = new RemoteStorage({
    client: new S3Client({
        credentials: {
            accessKeyId:process.env.S3_ACCESS_KEY,
            secretAccessKey:process.env.S3_SECRET_KEY
        },
        region:process.env.S3_REGION,
    }),
    params: {
        bucket:'mybucket'
    }
});

export { s3Storage }

router.js

import multer from 'multer';
import { s3Storage } from '../utilities/storage.js';
import {filter} from '../utilities/validators/fileValidator.js'; //Multer-related
import { topicValidation } from '../utilities/validators/middleware/validators.js'; //JOI

const router = express.Router({ caseSensitive: false, strict: false });

const s3Parser = multer({storage:s3Storage, fileFilter:filter, limits:{fileSize:1024000}});

router.route('/:username/create')
  .post(isLoggedIn,isAuthor,s3Parser.single('topic[file]'),topicValidation,createTopic)
Property Value
client S3Client Class
params bucket (required) and options for Upload class
options? See Below

The following data will be appended to req.file

variable data type info
bucket string
contentType string
etag string
filename string
metadata object metadata passed in Upload options
path string public URL
encryption string
versionId string undefined if versioning disabled

Options

All options are optional. Some may only apply to certain clients. See below.

Options Data Type Description
chunk_size number This will trigger a chunked upload for any client
public_id string or (req,file,cb) => string String or function that returns a string that will be used as the filename
trash string Alternative text for trash text. See Validation Note
validator (req,file,cb) => boolean See Validation
leavePartsOnError boolean S3 Only
queueSize number S3 Only
tags Tags[] S3 Only

Validation

Sometimes when uploading a file, it will be part of a form with other data that will exist on req.body. What if you need to validate that other form data before proceeding to upload the file? That's where this function comes in!

Here is an example:

/**
 * You will have access to Express's req.body
 * 
 * You can validate anything you wish and return true/false
 * on whether you want the file to upload (true = upload)
 * 
 * True or false will still cause the code to continue to the next 
 * middleware in your route. Calling cb(err) will result in Multer
 * calling next(err) in your app.
 * 
 * This example manually calls JOI's (a validation library for NodeJS)
 * validation function to validate the fields on req.body
 * 
 * In this example, you'll still need to call JOI's validator as a
 * middleware in your route AFTER multer to actually give the client
 * a response with error details
 * 
 * You can create a more complex function that checks on which
 * URL the client is visiting to decide which validator to use.
 * This prevents you from having to create another RemoteStorage
 * object for each type of validator
 */
const handleTopicValidation = (req, file, cb) => {
    let output = true;
        try {
            const {error, value} = topicValidator.validate(req.body);
            if (error) output = false;
        } catch(err) {
            output = false;
        }
        return output;
}

const topicStorage = new RemoteStorage({
    client:Cloudinary,
    params: {
        folder:'myfolder'
    },
    options: {
        chunk_size:1024 * 256 * 5,
        validator: handleTopicValidation
    },
});

Big Note on Validator

If you pass false, thus bypassing file upload, the software still has to pipe the readable stream somewhere. As a solution, it will use fs's createWriteStream function to create a file to dump the data in. Immediately afterwards, it will call fs's rm function to delete that file.

Two notes:

  1. Make sure your Node app has writing privileges in its own directory. Otherwise, you might get an Access Denied error.
  2. The default filename created is called trash.txt. You can use the trash option to customize the filename so it doesn't mess with any file you may have.

Public Id

The public_id option allows you to define a string, or a function that returns a string, to deal with naming the file you upload. Using this property may overwrite any similar function in the params object for the client you are using.

NOTE: Cloudinary does NOT want the file extension in the filename whereas Google Cloud Storage and AWS S3 do.

const handlePublicId = (req,file,cb) => {
    /*
        This example uses the date object to ensure a unique filename and assumes only
        one dot (prior to extension) exists
    */
    return `${file.originalname.split('.')[0]}-${Date.now()}.${file.originalname.split('.')[1]}`
}

const s3Storage = new RemoteStorage({
    client: new S3Client({
        credentials: {
            accessKeyId:process.env.S3_ACCESS_KEY,
            secretAccessKey:process.env.S3_SECRET_KEY
        },
        region:process.env.S3_REGION,
    }),
    params: {
        bucket:'mybucket'
    },
    options: {
        chunk_size: 1024 * 1000 * 10,
        public_id: handlePublicId
    }
});

TypeScript Example

import { RemoteStorage } from 'multer-remote-storage';
import { v2 as Cloudinary } from 'cloudinary';
import { CLOUDINARY_NAME, CLOUDINARY_API_KEY, CLOUDINARY_SECRET } from './config';
import topicValidator from './validators/topicValidator';

//Types
import { Request } from 'express';
import { File, MulterCallback } from 'multer-remote-storage';

const handleTopicValidation = (req: Request, file: File, cb: MulterCallback) => {
    let output = true;
    try {
        const { error, value } = topicValidator.validate(req.body);
        if (error) output = false;
    } catch (err) {
        output = false;
    }
    return output;
}

const handlePublicId = (req: Request, file: File, cb: MulterCallback) => {
    return `${file.originalname.split('.')[0]}-${Date.now()}}`
}

const storage = new RemoteStorage({
    client: Cloudinary,
    params: {
        folder: 'Programminghelp'
    },
    options: {
        public_id: handlePublicId,
        validator: handleTopicValidation
    },
});

export { Cloudinary, storage };

Package Sidebar

Install

npm i multer-remote-storage

Weekly Downloads

25

Version

0.0.6

License

ISC

Unpacked Size

51.4 kB

Total Files

13

Last publish

Collaborators

  • agadzinski93