Issue when decrypting a ReadStream created from s3 Object

[rsuh]

Hi all,

What I'm trying to accomplish is getting an encrypted object from s3, creating a stream from it, then decrypting that stream.

const obj = s3.getObject({
        Bucket: 'bucket',
        Key: 'encrypted_file',
}).createReadStream()

const decryptedStream = obj.pipe(decryptStream(keyring))

I've confirmed that I'm able to successfully perform this operation if the file is downloaded locally, however it seems that pulling from s3 there seems to be an issue.

The issue is:

Error: Unsupported state or unable to authenticate data
    at Decipheriv.final (internal/crypto/cipher.js:174:29)
    at DecipherStream._onAuthTag (/Users/rsuh/code/riotgames/event-bus-producer/node_modules/@aws-crypto/decrypt-node/src/decipher_stream.ts:177:29)

What I've noticed is that the entire file is not getting decrypted, just a portion of it is, and then it falls over.

As far as I've seen from googling this may have to do with encoding headers, however those fixes have not worked for me.

Thanks!

[rsuh]

Follow up:

So I tried piping from s3 to a local write file, and then reading from that file and piping that to decryptStream, and now I'm getting an error:

Error: Incomplete Header

When looking at the source code it mentions, A completed header MUST have been processed. Thus is there a possibility that s3 objects aren't returning the correct headers somehow? I don't believe its a data problem because downloading the file locally is fine

[seebees]

Hum, I'll have to dig into this.
Sounds like a timing issue.
Your are correct that it should "Just Work" the way you have composed it.

[rsuh]

Hm yea because the getObject call isn't a promise if we just createReadStream(). It seems that getObject returns a header and a body, we tried just decrypting the body but still no luck :/ Appreciate it!

[seebees]

So I tried this and I'm not getting the same error.
Here is what I did. Can you see how what I'm doing is different?
I'd like to get some repo of what you are doing to make sure it is not a bug :)

const generatorKeyId = 'kmsKeyArn'
const Bucket = 'yourBucketName'
const Key = 'encrypted_file'



import { encryptStream, decryptStream, KmsKeyringNode } from '@aws-crypto/client-node'
import { S3 } from 'aws-sdk'
import { createReadStream, createWriteStream } from 'fs'
import { pipeline } from 'stream'
import { promisify } from 'util'
const pipelinePromise = promisify(pipeline)

const keyring = new KmsKeyringNode({ generatorKeyId })
const s3 = new S3()

const context = {
  stage: 'demo',
  purpose: 'simple demonstration app',
  origin: 'us-west-2',
}

const readStream = createReadStream('/path/to/file/to/upload')
  .pipe(encryptStream(keyring, { encryptionContext: context}))

s3.upload({Bucket, Key, Body: readStream})
  .promise()
  .then(console.log)
  .then(() => {
    const s3ObjectStream = s3.getObject({Bucket, Key}).createReadStream()
    const transformStream = decryptStream(keyring)
      .on('MessageHeader', ({ encryptionContext }) => {
        Object.entries(context).forEach(([key, value]) => {
          if (encryptionContext[key] !== value)
            throw new Error('Encryption Context does not match expected values')
        })
      })
    const writeStream = createWriteStream('/path/to/file/you/want/to/save')
    return pipelinePromise(s3ObjectStream, transformStream, writeStream)
  })
  .then(console.log)

[rsuh]

I'll try this implementation out, but the one difference between our implementations is that for me, I am decrypting a s3 file that was encrypted using the java sdk, however nothing in the API points to that being an issue? Maybe the encryption context was set differently im not 100% sure, but yea that's the main difference

[seebees]

The Java AWS Encryption SDK and the JavaScript AWS Encryption SDK are compatible.
However there is also the AWS S3 Encryption Client inside the Java AWS SDK (no encryption).
These are unfortunately NOT the same and NOT interoperable.

On the Java side which one of these looks like what you are doing:
AWS S3 Encryption Client:
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-crypto.html

Java AWS Encryption SDK:
https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/java.html

[rsuh]

I believe its the Java AWS Encryption SDK

These are the imports in the java app

import com.amazonaws.encryptionsdk.AwsCrypto;
import com.amazonaws.encryptionsdk.CryptoAlgorithm;
import com.amazonaws.encryptionsdk.CryptoInputStream;
import com.amazonaws.encryptionsdk.kms.KmsMasterKey;
import com.amazonaws.encryptionsdk.kms.KmsMasterKeyProvider;

[seebees]

Have you tried doing both the reading and writing in Java?
I'm not sure how to get a reproducible case so
we can work out what is going on.

[rsuh]

Haha btw appreciate all the support here! We're going to tear down the java implementation and try a typescript implementation, will update the thread here if we run into anything funky there! Thanks :) I will close if resolved!

[seebees]

Ok, if you figure out what is going on, let us know.
Interoperability is hard but it is a priority for the AWS Encryption SDK :)

[pravink135]

Even i was also getting this error
Error: Unsupported state or unable to authenticate data
While downloading the file from S3 bucket & piping the response for further decryption. Is there any other alternate way of resolving this.

[seebees]

@pravink135 Do you have an example that I can use to reproduce this error?

[pravink135]

@seebees
Yes i was doing this way, correct me if i am doing any mistakes.

var stream = require('stream');
const request = require('request');
const util_1 = require("util");
const finishedAsync = util_1.promisify(stream.finished);
const client_node_1 = require("@aws-crypto/client-node");
let buff = [];

const context = {
stage: 'demo',
purpose: 'simple demonstration app',
origin: 'us-west-2',
}

async downloadDecrypt(){
const fileStream = request('s3 file url here')
.pipe(decryptStream(keyring), {encryptionContext: context})
.on('MessageHeader', ({ encryptionContext }: MessageHeader) => {
Object
.entries(context)
.forEach(([key, value]) => {
if (encryptionContext[key] !== value) throw new Error('Encryption Context does not match expected values')
})
})
fileStream.on('data',(chunk)=>{
buff.push(chunk);
});
await finishedAsync(fileStream);
fs.writeFileSync('./decryptedData.txt',Buffer.concat(buff).toString('utf-8'));
}

[seebees]

@pravink135 This won't help me recreate the error per se.
Because this only does the reading.
If you had an example of how you wrote the file to S3,
then I could try and repo it on my end :)

However, it looks like is should work.

That said, I would suggest a slight tweak.
You should be able to just stream the data right into the file.

async downloadDecrypt(){
  const fileStream = request('s3 file url here')
  .pipe(decryptStream(keyring), {encryptionContext: context})
  .on('MessageHeader', ({ encryptionContext }: MessageHeader) => {
    Object
    .entries(context)
    .forEach(([key, value]) => {
      if (encryptionContext[key] !== value) throw new Error('Encryption Context does not match expected values')
    })
  })
  .pipe(fs.createWriteStream('./decryptedData.txt', { encoding: 'utf8' }))
  return finishedAsync(fileStream);
}