I am Hack Sparrow
Captain of the Internets.

Node.js – Amazon S3: How to get Started

Getting started with Amazon S3 and Node.js

Believe it or not, integrating Amazon S3 service with Node.js is very easy. You just need to know two things to be able to use Amazon S3 and Node.js together smoothly.

The first thing is a commandline utility called s3cmd. s3cmd lets you create S3 buckets, upload files to them, delete files from buckets and delete buckets - from the commandline / terminal.

The second is a Node.js module called knox. Knox provides all (probably) the functionality that s3cmd has to offer, except that it integrates directly with Node.js as a module.

So why do you need two of them? Well, actually you can integrate your Node app with S3 using either of them, but when you use them together the development process is going to be a lot more pleasant. The gist is this: use knox of managing Amazing S3 buckets from your Node app - use s3cmd to monitor the changes from the commandline - during the development process. You could very well use knox to monitor the changes too, but using s3cmd is much easier.

Now let's get down to learning how to actually use s3cmd and knox.

Setting up s3cmd and how to use it

Download the latest source code of s3cmd from here. It was s3cmd-1.0.1.tar.gz for me. Untar and install it this way:

$ tar -zxvf s3cmd-1.0.1.tar.gz
...
$ cd s3cmd-1.0.1
$ python setup.py install
...
$ s3cmd --version
s3cmd version 1.0.1

Note: you may need to open a new instance of commandline if you don't see the version in the current instance.

Now it's time to configure s3cmd on your system. Be ready with your Access Key ID and Secret Access Key. You will be asked for GPG encryption key and GPG location too - they are optional. But if you are interested in encryption, refer the official s3cmd page.

$ s3cmd --configure

Some basic s3cmd commands

To list your S3 buckets:

$ s3cmd ls

To create an S3 bucket:

$ s3cmd mb s3://b1.buckethead

To see the content of the bucket s3://b1.buckethead

$ s3cmd ls s3://b1.buckethead

To upload files to s3://b1.buckethead:

$ s3cmd put slash.jpg s3://b1.buckethead/guitarists/slash.jpg 

Uploaded files are private by default. To make them public:

$ s3cmd put --acl-public petrucci.jpg s3://b1.buckethead/guitarists/petrucci.jpg 

Public files can be accessed using browsers thus: http://b1.buckethead.s3.amazonaws.com/guitarists/petrucci.jpg
I haven't really created that bucket or uploaded that file, so you won't get anything if you load the URL, create your own and try.

To download file from S3:

$ s3cmd get s3://b1.buckethead/guitarists/slash.jpg

To download file from S3 (save-as):

$ s3cmd get s3://b1.buckethead/guitarists/slash.jpg slash-s3.jpg

To delete a file:

$ s3cmd del s3://b1.buckethead/guitarists/slash.jpg

To delete a folder (and its contents):

$ s3cmd del --recursive --force s3://b1.buckethead/guitarists/

To delete a bucket:

$ s3cmd rb s3://b1.buckethead

So that's the basics of s3cmd and what all you can do on Amazon S3. For more details about s3cmd commands and options, visit the official page or read up the README file that came with the source code.

Now that you know how S3 works, you will get a better perspective of what to expect from knox and what all you might want to do.

Installing Node module Knox and how to use it

Installing knox is simple:

$ npm install knox

Once you have installed Knox, create an instance of the S3 client:

var S3_KEY = 'GJGA49HA90MPDRAH2FGAU';
var S3_SECRET = 'HJGAjgdh4387hHAhskhdjjFFGAjh24jhjDKKJHDgs3aH';
var S3_BUCKET = 'b1.buckethead';
var knox = require('knox').createClient({
key: S3_KEY,
secret: S3_SECRET,
bucket: S3_BUCKET
});

Note: use a valid S3 key and Secret. The one I have used above is junk, if it is not obvious to you.

To upload a file to S3 bucket:

knox.putFile('slash.jpg', 'guitarists/slash.jpg', {'Content-Type': 'image/jpeg'}, function(err, result) {
if (200 == result.statusCode) { console.log('Uploaded to mazon S3'); }
else { console.log('Failed to upload file to Amazon S3'); }
});

Not specifying the MIME type will result in the file being assumed as binary. Your images will be prompted for download, instead of displaying in the browser, so make sure you don't miss the Content-Type field.

All file uploads are public by default, to make it private pass an object with x-amz-acl set to private:

knox.putFile('gilbert.jpg', 'guitarists/gilbert.jpg', {'Content-Type': 'image/jpeg', 'x-amz-acl': 'private'}, function(err, result) {
if (200 == result.statusCode) { console.log('Uploaded to mazon S3'); }
else { console.log('Failed to upload file to Amazon S3'); }
});

To download file do this:

// We need the fs module so that we can write the stream to a file
var fs = require('fs');
// Set the file name for WriteStream
var file = fs.createWriteStream('slash-s3.jpg');
knox.getFile('slash.jpg', function(err, res) {
res.on('data', function(data) { file.write(data); });
res.on('end', function(chunk) { file.end(); });
});

To delete a file do this:

knox.deleteFile('/guitarists/slash.jpg', function(err, res) {
if (200 == result.statusCode) { console.log('Deleted'); }
else console.log('Failed to delete');
});

That is the basics of using Knox with Node. If you want to know more about other available options, visit its official GitHub page.

Note: s3cmd or knox, you need to escape the space character in your directory and file names, else your operations will fail.

Wrong

$ s3cmd put slash.jpg s3://b1.buckethead/amazing guitarists/slash.jpg 

Right

$ s3cmd put slash.jpg s3://b1.buckethead/amazing\ guitarists/slash.jpg 

Wrong

knox.putFile('slash.jpg', '/amazing guitarists/slash.jpg' ...

Right

knox.putFile('slash.jpg', '/amazing\\ guitarists/slash.jpg' ...

So that's all about connecting to S3 from Node.js. Pretty simple, isn't it? Any queries, need examples etc ... ping me in the comments.

9 Responses to “Node.js – Amazon S3: How to get Started”

  1. tani says:

    Is there a way to delete s3 folders (containing images) through knox module?

  2. Régis says:

    Hey,

    Thanks very much for you helpfull post.

    Have you tried this :
    var stream = fs.createReadStream(‘data.json’);
    client.putStream(stream, ‘/some-data.json’, function(err, res){
    // Logic
    });

    ?

    Regards,

    Régis

  3. Daniel says:

    Is there a way to create new buckets via javascript?

  4. Aykut says:

    I think deleteFile doesn’t return 200, expected response is 204:

    See:

    http://docs.amazonwebservices.com/AmazonS3/latest/RESTObjectDELETE.html

  5. Jasdeep Khalsa says:

    You are correct Aykut, the delete expected response is 204.

    The new link is here to S3 documentation is:

    http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectDELETE.html

  6. k srinivasulu says:

    I would like to upload a form directly to the amazon s3 using Nodejs. I am unable to get code for that one, so please share your information with me.

    Regards,
    srinivasulu

  7. Captain says:

    Use the official Node SDK from Amazon – http://aws.amazon.com/sdkfornodejs/

  8. Patrick D'appollonio says:

    Hello,

    What about upload an image from an user-uploaded image stream?

  9. Rinku says:

    Hi,

    I am a beginner for amazon webservices. Can you plz provide a solution for the following problem.
    Can I create folders dynamically and rename the file name by appending the current sysdate in S3 by copying the data from other bucket in S3 using S3 express.
    Kindly help asap.

    Thanks,
    Rinku

Make a Comment