Node.js Streams Explained: Readable, Writable, Transform, and Backpressure

What is Node.js streams module? Imagine you are building a backend application that accepts file uploads, processes documents, generates reports, and sends data over HTTP. Some files are small, but others may be several gigabytes in size.

How should your application handle them?

One approach is to read an entire file into memory, process it, and then write the result:

import * as fs from "node:fs/promises";

async function copyFile() {
  // Load the entire source file into a Buffer.
  const content = await fs.readFile("video.mp4");

  // Write the complete Buffer to a new file.
  await fs.writeFile("video-copy.mp4", content);

  console.log("File copied successfully.");
}

copyFile().catch(console.error);

This approach works, but readFile() returns the complete contents. If the file is several gigabytes, the application needs enough memory to hold that content. Concurrent operations can increase the demand even further.

There is another problem. Sometimes we do not need the complete file at once. If our task is to copy a file, compress it, calculate its checksum, or forward it to another destination, we can often process its contents incrementally.

Instead of reading the entire file first, we can read a chunk, process it, pass it along, and repeat until the input is exhausted.

That is the central idea behind Node.js Streams.

Streams provide an abstraction for handling data incrementally. They are useful for files, network connections, HTTP requests and responses, compression, and other operations in which data flows from a source to a destination.

In this tutorial, we will build the concept from scratch. We will understand how streams work, explore the four stream types, learn their essential methods and events, build custom transformations, and examine the buffering and backpressure mechanisms that make streaming effective.

The goal is not merely to memorize methods such as pipe(), write(), or end(). It is to understand how and when to use them.

1. The Fundamental Idea: Process Data as It Arrives

1.1 What happens when you read a file?

In our filesystem tutorial, we saw that a file is an external resource managed through Node.js and the operating system.

When we use fs.readFile(), Node.js reads the file contents and eventually returns them as a Buffer, or as a string if we request an encoding.

The application can then work with the complete result.

             File on Disk                  |
                  v
             Read Entire File
                  |
                  v
             Buffer in Memory
                  |
                  v
             Process the Data

This is an excellent approach when the file is small and the program needs all its contents together.

However, reading an entire file becomes less attractive when the file is large or when the data can be processed incrementally.

1.2 What changes when we use a stream?

A stream allows the application to process data in pieces called chunks.

              Large File                  |
                  v
            Read a Chunk
                  |
                  v
            Process Chunk
                  |
                  v
            Write Chunk
                  |
                  v
            Read Next Chunk
                  |
                  v
            Process Next Chunk
                  |
                 ...
                  |
                  v
               Finished

The important distinction is that processing can begin before the entire file has been read.

For a binary file, a chunk is commonly represented as a Buffer or Uint8Array. For a stream configured to decode text, chunks can instead be strings. Some streams operate in object mode and pass JavaScript objects rather than byte sequences.

As we learned when working with Buffers, bytes and strings are not the same thing. A Buffer represents bytes in memory, whereas a stream manages how data is delivered over time.

These concepts work together:

• A Buffer represents a piece of data in memory.
• A stream manages the incremental flow of data.
• A consumer processes the data.
• A destination accepts the resulting data.

1.3 Does streaming guarantee low memory usage?

Not automatically. A stream can buffer data, and poorly designed code can still accumulate unbounded amounts of it.

For example, this code reads a file incrementally but stores every chunk in an array:

import { createReadStream } from "node:fs";

async function readIntoArray(filePath) {
  const chunks = [];

  const source = createReadStream(filePath);

  for await (const chunk of source) {
    // Retain every chunk instead of discarding it after processing.
    chunks.push(chunk);
  }

  return Buffer.concat(chunks);
}

This eventually holds the complete file in memory, much like a full-file read.

Streaming helps control memory when the application processes chunks and releases its references instead of retaining all the data.

The real benefit comes from designing the data flow to remain incremental.

2. The Four Types of Node.js Streams

Node.js defines four fundamental stream types: Readable, Writable, Duplex, and Transform.

They describe the role a stream plays in a data flow.

2.1 Readable streams

A Readable stream provides data that a consumer can read.

Examples include a file being read from disk, an incoming HTTP request, or data received over a network connection.

       Source of Data              |
              v
       Readable Stream
              |
              v
           Consumer

A file-reading stream can be created with createReadStream():

import { createReadStream } from "node:fs";

const readable = createReadStream("notes.txt");

readable.on("data", (chunk) => {
  // This callback receives each available chunk.
  console.log("Received", chunk.length, "bytes");
});

readable.on("end", () => {
  // This runs when no more data is available.
  console.log("Finished reading.");
});

readable.on("error", (error) => {
  // Handle failures such as a missing or inaccessible file.
  console.error("Read failed:", error.message);
});

The data event is emitted when chunks are delivered to the listener. The end event signals that the readable stream has no more data to provide.

This example uses an event-based reading style. Later, we will compare it with async iteration and piping.

2.2 Writable streams

A Writable stream accepts data from a producer.

Examples include a file being written to disk, an outgoing HTTP response, or a socket sending data to another computer.

          Producer              |
              v
       Writable Stream
              |
              v
          Destination

Create one with createWriteStream():

import { createWriteStream } from "node:fs";

const writable = createWriteStream("output.txt");

writable.write("Hello ");
writable.write("Node.js");

// Tell the stream that no additional data will be written.
writable.end();

writable.on("finish", () => {
  console.log("Writing completed.");
});

writable.on("error", (error) => {
  console.error("Write failed:", error.message);
});

The resulting file contains:

Hello Node.js

The write() method submits data to the destination stream. The end() method tells it that no more data is coming.

The finish event indicates that all submitted data has been processed by the writable stream. It does not, by itself, guarantee that the data will survive every possible hardware failure or power loss.

One detail to remember: a Writable stream does not necessarily write to disk. A network socket and an HTTP response can also act as writable destinations.

2.3 Duplex streams

A Duplex stream supports both reading and writing.

         Data In
             |
             v
        +-----------+
        |   Duplex  |
        |   Stream  |
        +-----------+
             |
             v
          Data Out

A TCP socket is a common example. Your application can send data to a remote computer while receiving data from it through the same connection.

Its readable side and writable side operate independently. Sending data does not inherently mean that the same data will appear on the readable side.

Duplex streams are useful whenever a connection or component needs to support data in both directions.

2.4 Transform streams

A Transform stream is a specialized Duplex stream. It accepts input, transforms it, and produces output.

           Input             |
             v
      Transform Stream
             |
       Modify Data
             |
             v
           Output

Examples include compression, decompression, encryption, and text transformation.

For example, a compression pipeline could look like this:

       Readable File             |
             v
        Gzip Transform
             |
             v
      Writable File

The input data passes through the transform, and the destination receives compressed output.

We will implement a Transform stream later. For now, the key distinction is that a Duplex stream reads and writes, while a Transform stream reads, processes, and produces corresponding output.

These four types give us the vocabulary to describe most common stream architectures.

3. Creating File Streams

Node.js’s built-in node:fs module exposes methods for creating file streams.

The two main methods are:

import { createReadStream, createWriteStream } from "node:fs";

createReadStream() returns a Readable stream.

createWriteStream() returns a Writable stream.

They accept options that control how the file is accessed and how the stream behaves.

3.1 Understanding createReadStream()

The basic form is:

const readable = createReadStream("input.txt");

By default, the stream reads data as Buffers.

You can configure options such as the text encoding, the read range, and highWaterMark.

For example:

const readable = createReadStream("input.txt", {
  encoding: "utf8",
  highWaterMark: 1024,
});

Here, encoding: "utf8" means the stream decodes incoming bytes into strings.

The highWaterMark option sets the threshold used for the stream’s internal buffering. In this example, the threshold is 1024 bytes.

An important clarification: highWaterMark is not a strict guarantee that the stream or application will never use more than 1024 bytes of memory. It is a buffering threshold that influences when more data should be requested or when backpressure should be applied.

Node.js documents these options in its File System API.

3.2 Reading only part of a file

A read stream can also be configured with start and end offsets.

const readable = createReadStream("input.txt", {
  start: 0,
  end: 9,
});

These offsets are inclusive for the byte range. This stream requests bytes 0 through 9, which is a maximum of ten bytes, if those bytes exist.

This can be useful when working with files that contain fixed-format records, headers, or sections that can be read by position.

When using offsets, remember that they refer to bytes, not JavaScript string character positions.

3.3 Understanding createWriteStream()

The basic form is:

const writable = createWriteStream("output.txt");

A write stream creates the target file if necessary and normally truncates an existing file when it opens it for writing.

This default behavior is important: do not point a write stream at an existing file if you intend to preserve its contents.

A file can be appended to by opening a write stream with the append flag:

const writable = createWriteStream("application.log", {
  flags: "a",
});

Now the stream writes new data at the end of the file rather than truncating it.

You can also configure highWaterMark, encoding, a starting file position, and other options. Consult the official createWriteStream() documentation when you need particular file-opening behavior.

4. Understanding Readable Streams in More Depth

A Readable stream can be consumed in multiple ways. The method you choose affects when chunks are delivered and how much control you have over reading them.

Node.js supports two commonly discussed reading styles: flowing mode and paused mode.

4.1 Flowing mode

In flowing mode, the readable stream delivers chunks as data events.

For example:

import { createReadStream } from "node:fs";

const readable = createReadStream("notes.txt");

readable.on("data", (chunk) => {
  console.log(chunk);
});

Adding a data listener can put the stream into flowing mode.

This is convenient for simple consumption, but you must avoid doing substantial synchronous work inside the callback. A slow callback can delay other JavaScript operations, and an application that retains every chunk can still consume excessive memory.

4.2 Paused mode and read()

A readable stream can also be consumed using the readable event and the read() method.

import { createReadStream } from "node:fs";

const readable = createReadStream("notes.txt");

readable.on("readable", () => {
  // Read all chunks currently available.
  let chunk;

  while ((chunk = readable.read()) !== null) {
    console.log("Chunk:", chunk);
  }
});

The read() method retrieves data from the stream’s internal buffer. It returns null when no data is currently available for reading.

The readable event tells the application that data can be read. It does not necessarily mean that the complete file has arrived.

This approach is more manual than data events and can be useful when you need explicit control over when data is consumed.

For most application code, however, async iteration or pipeline() is often easier to reason about.

4.3 Don’t casually mix reading styles

A frequent source of confusion is registering data listeners, calling read(), and using pipe() on the same readable stream as if each were an independent consumer.

They are different ways of consuming the same stream. Switching between them without understanding the stream state can result in missed or unexpectedly consumed data.

Choose one reading style for a given stream unless you have a clear reason to coordinate them.

5. Readable Stream Methods: What They Do and When to Use Them

Here are the main Readable methods that application developers should recognize.

Method	Purpose
read([size])	Retrieves available data from the stream’s internal buffer.
setEncoding(encoding)	Makes the stream deliver decoded strings rather than raw byte Buffers.
pause()	Stops flowing-mode delivery temporarily.
resume()	Resumes flowing-mode delivery.
isPaused()	Reports whether the stream is currently paused in the flowing-mode sense.
pipe(destination)	Connects a readable source to a writable destination.
unpipe([destination])	Removes a pipe connection.
unshift(chunk)	Places data back into the readable buffer; an advanced operation.
wrap(oldStream)	Adapts certain older-style readable streams to the modern interface.
destroy([error])	Destroys the stream and optionally records an error.

A few other methods and properties are worth knowing.

readable.readableEnded indicates whether the stream has emitted end. readable.readableLength reports the amount of data currently buffered for reading, and readable.readableHighWaterMark exposes its configured buffering threshold.

readable.destroyed indicates whether the stream has been destroyed, while readable.errored exposes its recorded error when applicable.

These are useful for inspection and diagnostics, but they should not replace proper event handling or completion management.

5.1 What does setEncoding() do?

Suppose a file contains text:

const readable = createReadStream("notes.txt");

readable.setEncoding("utf8");

The stream now produces UTF-8-decoded strings instead of raw Buffers.

This also matters for multibyte characters. A UTF-8 character can span multiple bytes, and the underlying byte read might split those bytes across separate reads. The stream’s decoder handles those boundaries so a valid multibyte character is not incorrectly decoded one fragment at a time.

You can also pass the encoding through the createReadStream() options, as shown earlier.

5.2 What do pause() and resume() do?

pause() stops the stream from emitting additional data events in flowing mode until it is resumed.

resume() resumes that delivery.

These methods can be useful for manual flow control, but when connecting ordinary Readable and Writable streams, using pipe() or pipeline() is generally easier and safer because Node.js coordinates backpressure for you.

5.3 What does destroy() do?

destroy() marks a stream as destroyed and triggers its cleanup behavior. If an error is supplied, the stream’s error-handling behavior becomes relevant too.

You should not use it as a substitute for the ordinary completion mechanism. For example, a Readable normally signals the end of its data with end, whereas destroy() stops the stream prematurely.

When a stream is part of a pipeline, pipeline() coordinates stream errors and destruction across the participating stages.

6. Writable Stream Methods: Writing Data Correctly

Writable streams expose methods for submitting data, signaling completion, controlling buffering, and inspecting the stream’s state.

Method	Purpose
write(chunk[, encoding][, callback])	Submits a chunk to the writable stream.
end([chunk][, encoding][, callback])	Signals that no more data will be written, optionally submitting a final chunk.
setDefaultEncoding(encoding)	Changes the default encoding used for string writes.
cork()	Buffers writes rather than immediately forwarding each write to the underlying implementation.
uncork()	Releases data buffered by cork().
destroy([error])	Destroys the stream, optionally with an error.

The methods cork() and uncork() are useful in more specialized circumstances, such as grouping writes for implementations that support efficient batched writing. They are not normally required for simple file output.

6.1 The difference between write() and end()

Consider:

const writable = createWriteStream("output.txt");

writable.write("First line\n");
writable.write("Second line\n");

// No more data will be submitted after this call.
writable.end();

Calling write() submits data. Calling end() tells the stream that no more data is coming.

The stream may still have data buffered when end() is called. It will finish processing that data before emitting finish.

6.2 The callback passed to write()

The write() method optionally accepts a callback that is invoked when the submitted chunk has been handled by the stream:

writable.write("Hello\n", (error) => {
  if (error) {
    console.error("The write failed:", error.message);
    return;
  }

  console.log("This chunk was handled.");
});

The callback concerns that particular write. It does not mean the entire stream has finished.

For completion of the complete write sequence, use end() and coordinate with finish, or use an appropriate utility such as pipeline().

6.3 Writable stream state properties

Property	What it tells you
writable.writableLength	How much data is currently queued for writing.
writable.writableHighWaterMark	The configured threshold for write buffering.
writable.writableNeedDrain	Whether the stream currently requires a drain event before writing should continue.
writable.writableEnded	Whether end() has been called.
writable.writableFinished	Whether the stream has completed its write-side work.
writable.destroyed	Whether the stream has been destroyed.

These properties help with debugging and flow control. They are not substitutes for listening to the proper lifecycle events or handling errors.

7. Stream Events and Their Meanings

Streams are EventEmitters, so they expose events that report lifecycle changes and data availability.

The most important events include:

Event	Commonly used with	Meaning
data	Readable	A data chunk is being delivered.
readable	Readable	Data is available to retrieve with read().
end	Readable	No more readable data will be provided.
finish	Writable	Submitted write data has been processed and the writable side has finished.
drain	Writable	The writable stream can accept more data after backpressure.
error	Both	The stream reports an error.
close	Both	The stream or its underlying resource has closed, when applicable.
pipe	Writable	A readable stream has been connected to the writable destination.
unpipe	Writable	A readable stream has been disconnected from the destination.
pause	Readable	Flowing-mode delivery has paused.
resume	Readable	Flowing-mode delivery has resumed.

These events describe different things.

For example, the readable stream’s end event does not guarantee that a separate writable destination has finished writing its buffered data. Likewise, a writable stream’s finish event does not mean a complete application-level transaction has been verified.

For a multi-stage flow, a Promise-based pipeline is often preferable because it provides a single completion or failure result.

The Node.js Streams API documentation describes stream events, methods, properties, and lifecycle behavior in detail.

8. Piping Streams Together

We have already learned that a Readable stream produces data and a Writable stream accepts it. The next question is: how do we connect them?

Node.js provides the pipe() method.

import { createReadStream, createWriteStream } from "node:fs";

const readable = createReadStream("input.txt");
const writable = createWriteStream("output.txt");

// Connect the source to the destination.
readable.pipe(writable);

The flow is:

       input.txt           |
           v
    Readable Stream
           |
        Chunks
           |
           v
    Writable Stream
           |
           v
       output.txt

The readable stream supplies chunks, and pipe() forwards them to the writable stream. It also coordinates flow control so that a fast source does not indefinitely overwhelm the destination.

For simple applications, pipe() is convenient. However, when a pipeline needs reliable error propagation and a way to await completion, Node.js provides a more robust option: pipeline().

8.1 pipe() versus pipeline()

The two methods are related but serve slightly different purposes.

Feature	pipe()	pipeline()
Connects streams	Yes	Yes
Handles backpressure	Yes	Yes
Coordinates errors across the pipeline	Requires more care	Yes
Provides a convenient Promise-based completion result	No	Yes, through node:stream/promises
Useful for multi-stage processing	Yes, with careful handling	Often the preferred approach

For example:

import { createReadStream, createWriteStream } from "node:fs";
import { pipeline } from "node:stream/promises";

async function copyFile() {
  await pipeline(
    createReadStream("input.txt"),
    createWriteStream("output.txt"),
  );

  // This runs after the complete pipeline succeeds.
  console.log("Copy completed.");
}

copyFile().catch((error) => {
  console.error("Copy failed:", error.message);
});

Notice that we do not separately register a finish listener to determine whether the whole pipeline completed. The Promise returned by pipeline() resolves on success and rejects when the pipeline fails.

For most application code that connects multiple streams, pipeline() is a good default.

9. Transform Streams in Practice

A Transform stream is useful when data needs to be changed while it flows from a source to a destination.

Common examples include compression, decompression, encryption, data validation, and converting text between formats.

The architecture looks like this:

     Input Source
            |
            v
      Readable Stream
            |
            v
     Transform Stream
            |
            v
      Writable Stream
            |
            v
      Output Destination

Let’s implement our own Transform stream to convert text to uppercase.

9.1 Creating a custom Transform

import { createReadStream, createWriteStream } from "node:fs";
import { Transform } from "node:stream";
import { pipeline } from "node:stream/promises";

const uppercaseTransform = new Transform({
  // This method is called for incoming chunks.
  transform(chunk, encoding, callback) {
    try {
      // The source is configured to provide UTF-8 strings.
      const uppercase = chunk.toUpperCase();

      // Pass the transformed chunk to the readable side.
      callback(null, uppercase);
    } catch (error) {
      // Report the failure to the stream machinery.
      callback(error);
    }
  },
});

async function convertFile() {
  const source = createReadStream("input.txt", {
    encoding: "utf8",
  });

  const destination = createWriteStream("output.txt");

  // The transform sits between the source and destination.
  await pipeline(source, uppercaseTransform, destination);

  console.log("Transformation completed.");
}

convertFile().catch(console.error);

Suppose input.txt contains:

node.js streams
process data incrementally

The resulting output.txt contains:

NODE.JS STREAMS
PROCESS DATA INCREMENTALLY

9.2 Understanding transform() and callback()

The transform() method is the hook through which a custom Transform processes incoming chunks.

Its callback takes the form:

callback(error, outputChunk);

Passing null as the first argument indicates success. The second argument is the output produced from the input chunk.

If processing fails, pass the error to the callback.

For a transformation that produces output for every input chunk, this is enough. More complicated transformations may need to maintain state or produce additional output when the input ends.

One important distinction: the transform() method is an implementation hook supplied when defining the Transform. It is not a method that application code should call directly for each chunk. The stream machinery invokes it as data arrives.

9.3 Implementing _flush()

Sometimes a Transform must produce output after the source has finished providing input.

For example, a parser may need to handle a final incomplete record, or a format converter may need to write a footer.

The _flush() implementation hook exists for that purpose.

import { Transform } from "node:stream";

const exampleTransform = new Transform({
  transform(chunk, encoding, callback) {
    // Forward each input chunk unchanged.
    callback(null, chunk);
  },

  flush(callback) {
    // This hook runs when no more input remains.
    // It can produce additional final output.
    callback(null, "\n-- END OF DATA --\n");
  },
});

If this Transform receives the text Hello, the output also contains the final marker:

Hello
-- END OF DATA --

The flush() hook is optional. It is useful when the transformation requires finalization after all input chunks have been processed.

The documented method is _flush() when implementing a Transform by subclassing. When using the constructor options object, the corresponding option is named flush, as shown above.

10. Built-In Transform Streams: Compression and Decompression

Node.js includes useful Transform streams, so you do not need to implement every transformation yourself.

The node:zlib module, for example, provides compression and decompression streams.

10.1 Compress a file with Gzip

import { createReadStream, createWriteStream } from "node:fs";
import { createGzip } from "node:zlib";
import { pipeline } from "node:stream/promises";

async function compressFile() {
  await pipeline(
    createReadStream("input.txt"),
    createGzip(),
    createWriteStream("input.txt.gz"),
  );

  console.log("Compression completed.");
}

compressFile().catch(console.error);

The flow is:

         input.txt             |
             v
       Readable Stream
             |
             v
        Gzip Transform
             |
             v
       Writable Stream
             |
             v
        input.txt.gz

The file input.txt.gz contains compressed data, not ordinary readable text.

10.2 Decompress the file

import { createReadStream, createWriteStream } from "node:fs";
import { createGunzip } from "node:zlib";
import { pipeline } from "node:stream/promises";

async function decompressFile() {
  await pipeline(
    createReadStream("input.txt.gz"),
    createGunzip(),
    createWriteStream("restored.txt"),
  );

  console.log("Decompression completed.");
}

decompressFile().catch(console.error);

The output file restored.txt contains the original uncompressed text.

These examples show why the Transform abstraction is useful. Compression and decompression fit into the same pipeline architecture as a custom uppercase transformation.

You can combine several transforms in one pipeline, provided their input and output formats are compatible.

11. Backpressure: The Most Important Streaming Concept

So far, we have talked about chunks flowing through streams. But there is an important problem to solve.

What happens if a producer generates data faster than a consumer can process it?

Imagine that a readable file stream can supply chunks quickly, but the writable destination is a slow network connection.

Fast Producer                     Slow Consumer

  Chunk 1 --------------------------> Processing
  Chunk 2 --------------------------> Processing
  Chunk 3 --------------------------> Still busy
  Chunk 4 --------------------------> Still busy
  Chunk 5 --------------------------> Still busy

If the producer keeps submitting data without considering the consumer’s speed, queued data can accumulate in memory.

This is where backpressure comes in.

Backpressure is a way for the receiving side to signal that the producer should slow down until the receiver can accept more data.

A useful analogy is a water tank connected to a narrow pipe. If water flows into the tank faster than the pipe can carry it away, the tank fills up. A control mechanism must reduce the incoming flow to avoid overflow.

11.1 The role of highWaterMark

Each ordinary stream has a highWaterMark setting that influences how much data it buffers before backpressure is applied.

For example:

const writable = createWriteStream("output.txt", {
  highWaterMark: 1024,
});

Here, the writable stream’s buffer threshold is 1024 bytes.

However, highWaterMark is a threshold, not a strict memory limit. A stream can temporarily hold data beyond that threshold, and an application can consume much more memory through other buffers or retained references.

The important purpose is to help streams regulate data flow when producers and consumers operate at different speeds.

11.2 Understanding the return value of write()

When you call write(), Node.js returns a Boolean value.

const canContinue = writable.write(chunk);

If it returns true, the stream’s buffering state currently permits the producer to continue writing.

If it returns false, the producer should stop submitting further chunks until the writable stream emits drain.

Let’s see the control flow:

             writable.write(chunk)                       |
                       v
              Did it return true?
                 /          \
               Yes           No
                |             |
                v             v
         Continue writing   Pause writing
                              |
                              v
                       Wait for "drain"
                              |
                              v
                       Resume writing

11.3 Handling backpressure manually

Here is a simplified example that shows how drain works:

import { createWriteStream } from "node:fs";
import { once } from "node:events";

async function writeChunks() {
  const writable = createWriteStream("output.txt");

  try {
    for (let i = 0; i < 1000; i++) {
      const chunk = `Record ${i}\n`;

      // Submit the next chunk.
      const canContinue = writable.write(chunk);

      if (!canContinue) {
        // Wait until the stream has capacity again.
        await once(writable, "drain");
      }
    }

    // Tell the destination that no more data is coming.
    const finished = once(writable, "finish");
    writable.end();

    // Wait for all submitted data to finish writing.
    await finished;

    console.log("All records written.");
  } catch (error) {
    writable.destroy(error);
    throw error;
  }
}

writeChunks().catch(console.error);

The example demonstrates that the producer pauses when write() returns false and resumes only after the drain event.

For production applications, error handling around manual writes requires care because a stream can fail at different points during the operation. When you can express the task as a readable source, a destination, and optional transforms, using pipeline() is generally simpler.

11.4 How does pipe() manage backpressure?

When you connect streams with pipe(), Node.js coordinates the flow between the readable and writable sides.

If the writable stream starts applying backpressure, the readable stream is paused until the destination can accept more data.

This prevents a fast producer from continuously pushing data into an overloaded destination.

It is one reason streams are so effective for large files and network communication.

For more details, see the official Node.js documentation on buffering and backpressure.

12. Object Mode: Streams Do Not Have to Carry Bytes

So far, our examples have mostly used Buffers or strings.

Node.js streams can also operate in object mode, in which individual chunks represent JavaScript values rather than byte sequences.

This is useful when processing records, events, parsed messages, or other structured data.

Consider:

import { Readable } from "node:stream";

const source = Readable.from([
  { id: 1, name: "Amritesh" },
  { id: 2, name: "Riya" },
]);

for await (const user of source) {
  console.log(user.id, user.name);
}

Output:

1 Amritesh
2 Riya

The stream provides objects rather than raw byte Buffers.

Object mode is useful for application pipelines in which each chunk represents one logical item.

An important distinction is that highWaterMark in byte mode is measured in bytes, while in object mode it counts objects. That means the same numerical threshold can have very different memory implications depending on the stream’s mode and the size of each object.

13. Creating a Readable Stream from an Iterable

Sometimes your source is not a file or socket. It may be an array, generator, or asynchronous iterator that produces data.

Node.js provides Readable.from() to turn an iterable or async iterable into a Readable stream.

import { Readable } from "node:stream";

const source = Readable.from(
  ["Hello ", "from ", "a ", "stream!"],
  { objectMode: false },
);

for await (const chunk of source) {
  // Each chunk is available as the consumer iterates.
  process.stdout.write(chunk);
}

Output:

Hello from a stream!

This is useful when you already have a data-producing function and want to connect it to the stream ecosystem.

You can also create a Readable from an async generator, which is useful when data is produced over time:

import { Readable } from "node:stream";

async function* generateRecords() {
  for (let i = 1; i <= 3; i++) {
    // Simulate asynchronous work before producing each record.
    await Promise.resolve();

    yield `Record ${i}\n`;
  }
}

const source = Readable.from(generateRecords(), {
  objectMode: false,
});

for await (const chunk of source) {
  process.stdout.write(chunk);
}

Output:

Record 1
Record 2
Record 3

This illustrates another source of stream data: an asynchronous generator can produce chunks incrementally instead of constructing the entire output first.

14. Writable Streams in More Detail: cork() and uncork()

Writable streams support a few additional methods that are useful for specialized workloads.

cork() temporarily buffers writes so that they can be processed together. uncork() releases those buffered writes.

import { createWriteStream } from "node:fs";

const writable = createWriteStream("output.txt");

// Temporarily buffer submitted writes.
writable.cork();

writable.write("First ");
writable.write("second ");
writable.write("third");

// Release the buffered writes.
writable.uncork();

// Finish the stream.
writable.end();

The output file contains:

First second third

For many destinations, corking can help group writes more efficiently. Its usefulness depends on the destination’s implementation, and it is rarely necessary for ordinary file copying with pipeline().

The setDefaultEncoding() method can also change how string chunks are encoded when no encoding is supplied to write().

writable.setDefaultEncoding("utf8");

This affects string-writing behavior. It does not convert arbitrary Buffer contents into text.

For most applications, the default UTF-8 encoding is appropriate. Use an explicit encoding when your data format requires it.

15. Duplex and Transform Implementation Hooks

So far, we have mostly consumed existing streams or supplied a Transform callback. Node.js also lets you create custom stream classes by extending the underlying stream types.

This is a more advanced topic, but it is useful to understand the distinction between the public methods used by consumers and the internal hooks used to implement streams.

15.1 Implementing a custom Readable

A custom Readable implements _read() to produce data when the stream requests it. The implementation uses push() to make chunks available to consumers.

import { Readable } from "node:stream";

class NumberStream extends Readable {
  private current = 1;

  _read() {
    if (this.current > 3) {
      // null signals that no more data will be produced.
      this.push(null);
      return;
    }

    // Provide the next item as a string chunk.
    this.push(`${this.current}\n`);
    this.current++;
  }
}

const source = new NumberStream();

source.pipe(process.stdout);

Output:

1
2
3

The important idea is that _read() is an implementation hook. Consumers do not normally call it themselves. The stream machinery asks for more data, and the custom implementation supplies it.

15.2 Implementing a custom Writable

A custom Writable implements _write() to decide what happens when the stream receives a chunk.

import { Writable } from "node:stream";

const destination = new Writable({
  write(chunk, encoding, callback) {
    // Convert the incoming chunk to text for this example.
    console.log("Received:", chunk.toString());

    // Signal that processing of this chunk has completed.
    callback();
  },
});

destination.write("Hello\n");
destination.end();

Output:

Received: Hello

The _write() implementation hook is where a custom destination processes each chunk. Calling the callback signals that the chunk has been handled, or reports an error if processing failed.

For a real asynchronous destination, such as a database adapter, this callback must be invoked only after the write operation has completed or failed.

15.3 Important hooks for stream implementers

When implementing custom stream classes, these hooks are worth recognizing:

Hook	Stream type	Purpose
_construct()	Readable or Writable implementations	Perform initialization before normal stream operations begin.
_read(size)	Readable	Produce or request more data.
_write(chunk, encoding, callback)	Writable	Process an incoming chunk.
_writev(chunks, callback)	Writable	Optionally process several queued writes as a batch.
_final(callback)	Writable	Perform final work before the writable side finishes.
_transform(chunk, encoding, callback)	Transform	Convert input chunks into output chunks.
_flush(callback)	Transform	Produce any final output after input ends.
_destroy(error, callback)	Custom stream implementations	Release implementation-specific resources during destruction.

These methods have underscore prefixes because they are implementation hooks. Application code should generally use the public APIs such as write(), end(), pipe(), and destroy(), rather than invoking the hooks directly.

The official documentation has dedicated guidance for implementing Readable streams, implementing Writable streams, and implementing Transform streams.

16. Other Useful Stream Utilities

In addition to the methods on individual stream objects, Node.js provides several utility functions.

16.1 pipeline()

We have already used pipeline() throughout the tutorial.

It connects a sequence of streams, coordinates backpressure, and provides error handling across the pipeline. The version in node:stream/promises returns a Promise, making it convenient with async and await.

16.2 finished()

The finished() utility lets you detect when a stream has completed or failed.

It is useful when you need to monitor an existing stream independently of connecting a whole pipeline.

import { createReadStream } from "node:fs";
import { finished } from "node:stream/promises";

const readable = createReadStream("input.txt");

try {
  // Wait for the readable stream to complete.
  await finished(readable);
  console.log("Stream finished.");
} catch (error) {
  console.error("Stream failed:", error.message);
}

This example waits for the stream to finish, but it does not consume the stream’s data. In practice, you must also establish an appropriate consumption method when data is expected.

For most source-to-destination processing, pipeline() is the more natural choice.

16.3 PassThrough

A PassThrough stream is a Transform stream that passes input data through without modifying it.

It can be useful for observing data, introducing a stream boundary, or inserting instrumentation into a pipeline.

import { PassThrough } from "node:stream";

const passThrough = new PassThrough();

passThrough.on("data", (chunk) => {
  console.log("Observed:", chunk.toString());
});

passThrough.end("Hello streams");

Output:

Observed: Hello streams

16.4 Readable.from() and stream composition

We have seen that Readable.from() can turn an iterable or async iterable into a Readable stream.

Node.js also provides compose() methods and utilities for combining streams into larger streaming operations. These can be useful when building reusable components or exposing one combined stream interface around several processing steps.

There are also interoperability utilities for converting between Node.js streams and Web Streams. These matter when integrating Node.js stream code with Fetch APIs or other APIs based on the Web Streams standard.

For a detailed reference to these APIs, consult the Node.js Streams documentation. Availability and stability can vary by Node.js version, so check the minimum runtime version before depending on newer convenience methods.

17. Error Handling and Cancellation

A stream pipeline can fail at several stages: the file might not exist, reading might fail, transformation might throw an error, or writing might fail.

A robust application needs a strategy for handling those failures.

When using pipeline(), failures are propagated through the returned Promise, making them easier to handle in one place:

import { createReadStream, createWriteStream } from "node:fs";
import { pipeline } from "node:stream/promises";

async function copyFile(sourcePath, targetPath) {
  try {
    await pipeline(
      createReadStream(sourcePath),
      createWriteStream(targetPath),
    );

    console.log("Copy succeeded.");
  } catch (error) {
    console.error("Copy failed:", error.message);
  }
}

copyFile("missing.txt", "output.txt");

If the source cannot be opened, the pipeline fails and the error is caught. The same approach handles errors from other participating stream stages.

Cancelling a pipeline

Some operations should be cancellable, such as a user abandoning an upload or a server shutting down a long-running task.

The Promise-based pipeline() supports an AbortSignal:

import { createReadStream, createWriteStream } from "node:fs";
import { pipeline } from "node:stream/promises";

const controller = new AbortController();

async function copyWithCancellation() {
  try {
    await pipeline(
      createReadStream("large-file.bin"),
      createWriteStream("copy.bin"),
      { signal: controller.signal },
    );

    console.log("Copy completed.");
  } catch (error) {
    console.error("Copy stopped:", error.message);
  }
}

// In a real application, call this when cancellation is requested:
// controller.abort();

copyWithCancellation();

The comment shows where cancellation would be triggered. Since the example does not actually call abort(), it runs normally unless another failure occurs.

When cancellation occurs, the pipeline rejects and the associated stream stages are destroyed as appropriate. Cancellation behavior should be part of the application design whenever long-running stream operations need to stop safely.

18. A Practical Project: Compress a File Using a Pipeline

Let’s combine the concepts into a practical utility that compresses a text file.

Create a file called compress-file.mjs:

import { createReadStream, createWriteStream } from "node:fs";
import { createGzip } from "node:zlib";
import { pipeline } from "node:stream/promises";

async function compressFile(inputPath, outputPath) {
  // Read the source as bytes, without decoding it as text.
  const source = createReadStream(inputPath);

  // Compress chunks incrementally using Gzip.
  const gzip = createGzip();

  // Write the compressed chunks to the destination.
  const destination = createWriteStream(outputPath);

  // Coordinate data flow, backpressure, errors, and completion.
  await pipeline(source, gzip, destination);

  console.log("Compression completed.");
}

compressFile("input.txt", "input.txt.gz").catch((error) => {
  console.error("Compression failed:", error.message);
});

Run the program:

node compress-file.mjs

On success, the output is:

Compression completed.

The file input.txt.gz contains the compressed representation of the source file.

The pipeline is composed of three stages:

        input.txt
              |
              v
       Readable Stream
              |
              v
         Gzip Transform
              |
              v
       Writable Stream
              |
              v
         input.txt.gz

The file is read incrementally, compressed as chunks pass through the transform, and written incrementally to the destination. There is no need to first load the complete source file into one Buffer.

This is the type of problem that streams are designed to solve.

19. When Should You Use readFile() Instead of Streams?

Streams are valuable, but they are not automatically the best choice for every file operation.

Use readFile() when the file is small enough to fit comfortably in memory and the application needs its complete contents together, such as a small configuration file.

Use streams when the file may be large, data arrives progressively, transformations can be applied incrementally, or a source needs to feed directly into a destination.

For example:

Requirement	Suitable approach
Read a small configuration file	readFile()
Copy a large video	Readable-to-Writable pipeline
Compress a large file	pipeline() with a Gzip transform
Process records incrementally	Readable stream or async iteration
Send data through a network socket	Streams and network APIs
Build a custom data transformation	Transform stream

The important question is whether the program needs the entire content at once or can do its work incrementally.

Conclusion: Streams Are About Managing Data Flow

Node.js Streams provide a standard way to handle data incrementally, making them useful for file processing, networking, compression, and other tasks that move data between sources and destinations.

We started with the limitations of loading entire files into memory, then explored the four fundamental stream types, file-stream creation, readable and writable methods, lifecycle events, and the role of Buffer chunks.

We then connected streams with pipe() and pipeline(), implemented a Transform, explored backpressure, examined object mode and async iteration, and reviewed stream implementation hooks and utilities.

The most important engineering insight is that streaming is not merely about reading smaller pieces of a file. It is about coordinating data production, buffering, consumption, errors, and completion.

Once you understand those responsibilities, stream APIs become easier to reason about, and you can choose between a Buffer, a full-file operation, and a streaming pipeline based on the actual needs of the application.

Official Documentation and Further Reading

• Node.js Streams API
• Node.js File System API
• Node.js Stream Promises API
• Node.js Buffer API
• Node.js HTTP API
• Node.js Zlib API