Node.js Buffer Explained: Bytes, Binary Data, and Character Encoding

In this tutorial we will learn about Node.Js Buffer in depth. While working with files, we encountered an important detail: when Node.js reads a file without specifying an encoding, it returns a Buffer rather than a regular JavaScript string.

Why?

Because files contain bytes, and a string is a representation of text. They are not the same thing.

A text file contains bytes that can be interpreted using a character encoding. An image contains bytes that describe image data. A PDF contains bytes arranged according to the PDF format. A network message might contain headers, identifiers, and a payload represented as binary data.

To work with these different kinds of data, Node.js provides the Buffer class.

Today, we will build our understanding from the ground up. We will learn what a byte is, how computers represent text using bytes, why character encoding matters, how to create and manipulate Buffers, and how Buffers are used with files and network data.

By the end, you should understand not just which Buffer method to call, but also what the method does to the underlying data.

Part 1: Before Buffer, Understand Bytes

1. What Is a Bit?

Computers represent information using bits. A bit has one of two values:

0 or 1

A bit is the smallest unit of binary information.

Individual bits can be combined to represent larger values. For example, using eight bits, we can represent values from 0 through 255 when interpreting those bits as an unsigned integer.

Bit positions:

128  64  32  16   8   4   2   1
  |   |   |   |   |   |   |   |
  0   1   0   0   0   0   0   1

In this example, the bits with values 64 and 1 are set to 1.

Therefore:

[
64 + 1 = 65
]

The binary sequence 01000001 represents the decimal number 65.

2. What Is a Byte?

A byte consists of eight bits.

1 byte = 8 bits

An unsigned byte can represent 256 possible values

This is a fundamental building block of digital data.

For example:

Decimal	Binary	Hexadecimal
0	00000000	00
1	00000001	01
10	00001010	0A
65	01000001	41
255	11111111	FF

The same byte can be written in different numerical representations. Decimal, binary, and hexadecimal are different ways of expressing the same underlying value.

This matters in Node.js because a Buffer stores byte values. When we print a Buffer, Node.js often displays its contents in hexadecimal notation.

For example:

import { Buffer } from "node:buffer";

const data = Buffer.from([72, 105]);

console.log(data);

Output:

<Buffer 48 69>

Why does the output show 48 69 instead of 72 105?

Because 72 in decimal is 48 in hexadecimal, and 105 in decimal is 69 in hexadecimal.

The underlying bytes have not changed. Only their displayed representation is different.

Part 2: How Does Text Become Bytes?

Now let us connect bytes to something familiar: text.

Consider this JavaScript statement:

const message = "Hi";

When JavaScript works with message, it treats it as a string.

But suppose we want to save that message to a file or send it across a network connection. Those operations require the string to be represented as bytes.

How does the computer decide which bytes represent the letters H and i?

The answer is character encoding.

A character encoding defines how text is represented as bytes and how those bytes are interpreted as text.

1. ASCII: A Simple Starting Point

ASCII is an older character-encoding standard that assigns numerical values to a set of characters, including English letters, digits, and common symbols.

For example:

Character H -> Decimal 72 -> Hex 48
Character i -> Decimal 105 -> Hex 69

Therefore, the text:

Hi

can be represented by these two bytes in ASCII-compatible encoding:

48 69

This is a useful first example because each character in this text occupies one byte in ASCII and UTF-8.

However, not every character can be represented using a single byte.

What about a symbol such as €, an Indian currency symbol such as ₹, or an emoji?

That is where UTF-8 becomes important.

2. UTF-8: Representing Text Across Different Languages

UTF-8 is a variable-length Unicode encoding. It represents Unicode code points using between one and four bytes.

For example:

Text:     A
UTF-8:    41
Bytes:    1

The uppercase letter A is represented by one byte.

Now consider the euro symbol:

Text:     €
UTF-8:    E2 82 AC
Bytes:    3

And consider an emoji:

Text:     😀
UTF-8:    F0 9F 98 80
Bytes:    4

The important lesson is that the number of characters in a string is not necessarily equal to the number of bytes required to encode it.

This distinction becomes critical when reading files, constructing binary messages, calculating payload sizes, or working with network protocols.

3. A Character Is Not Necessarily One Byte

Let’s test this in Node.js:

import { Buffer } from "node:buffer";

const examples = ["A", "€", "₹", "😀"];

for (const text of examples) {
  console.log({
    text,
    stringLength: text.length,
    byteLength: Buffer.byteLength(text, "utf8"),
    bytes: Buffer.from(text, "utf8"),
  });
}

The important results are:

Text	JavaScript string.length	UTF-8 byte length
A	1	1
€	1	3
₹	1	3
😀	2	4

Why does 😀 have a JavaScript string length of 2?

JavaScript strings use UTF-16 code units internally. The emoji is represented by a surrogate pair, so string.length returns 2. Its UTF-8 representation, however, requires four bytes.

That is why these operations serve different purposes:

text.length

Measures the number of UTF-16 code units in the JavaScript string.

Buffer.byteLength(text, "utf8")

Calculates how many bytes the string needs when encoded as UTF-8.

When dealing with file sizes or network payload lengths, byte length is often the measurement you need.

Part 3: What Exactly Is a Node.js Buffer?

A Buffer is a Node.js class for working with sequences of bytes.

It is built on top of JavaScript’s Uint8Array typed-array abstraction and provides methods that make binary data easier to create, inspect, convert, and manipulate.

Think of it as a sequence of byte-sized values stored in memory.

Buffer containing "Hi" in UTF-8

Index:      0       1
         +-------+-------+
Value:   |  0x48 |  0x69 |
         +-------+-------+
            H       i

The Buffer doesn’t inherently know that the bytes represent the letters H and i. It simply stores the byte values.

When we call a method such as toString("utf8"), we ask Node.js to interpret those bytes using UTF-8 decoding.

This distinction is useful because the same Buffer can contain text, binary values, or a mixture of both.

Why Doesn’t Node.js Just Use Strings for Everything?

Strings are appropriate when the data represents text.

But consider an image. Its contents might include pixel values, compression data, file-format headers, and other binary structures. Treating arbitrary bytes as text can alter their interpretation or produce replacement characters during decoding.

Similarly, a network packet might contain a numeric length field followed by a payload. The application needs to preserve and interpret those bytes according to the protocol, not assume that everything is text.

The Buffer provides a suitable representation for these situations.

When Will You Encounter Buffers?

You will commonly encounter Buffers when reading files without an encoding, working with TCP connections, processing binary file formats, handling compressed data, converting between encodings, and interacting with APIs that return binary payloads.

Node.js’s file-system APIs return a Buffer when no text encoding is specified. Supplying an encoding such as "utf8" asks Node.js to decode the bytes into a string instead. See the official Node.js File System documentation.

Part 4: Creating Buffers

Let’s explore the three creation techniques you should understand first.

1. Buffer.from(): Create a Buffer from Existing Data

When you already have data and want to represent it as a Buffer, Buffer.from() is often the method you need.

Create a Buffer from a string:

import { Buffer } from "node:buffer";

const buffer = Buffer.from("Hello", "utf8");

console.log(buffer);
console.log(buffer.toString("utf8"));

Output:

<Buffer 48 65 6c 6c 6f>
Hello

The string is encoded as UTF-8 bytes, and the Buffer stores those bytes.

You can also create a Buffer from an array of byte values:

const buffer = Buffer.from([72, 101, 108, 108, 111]);

console.log(buffer.toString("utf8"));

Output:

Hello

Each number represents one byte.

You can also create a Buffer from hexadecimal data:

const buffer = Buffer.from("48656c6c6f", "hex");

console.log(buffer.toString("utf8"));

Output:

Hello

Here, the hexadecimal string represents the byte sequence for Hello.

Use Buffer.from() when you already have a string, an array of bytes, another Buffer, or another supported input that you want to represent as binary data.

2. Buffer.alloc(): Allocate a Buffer Safely

Sometimes you need a buffer with a specific capacity before you have the data to put into it.

For example:

const buffer = Buffer.alloc(5);

console.log(buffer);

Output:

<Buffer 00 00 00 00 00>

Buffer.alloc(5) creates a Buffer containing five zero-filled bytes.

Why is it initialized to zero?

Because Buffer.alloc() is designed to give you a buffer whose contents are initialized rather than exposing whatever data happened to occupy the allocated memory previously.

You can also provide an initial fill value:

const buffer = Buffer.alloc(5, 65);

console.log(buffer);
console.log(buffer.toString("utf8"));

Output:

<Buffer 41 41 41 41 41>
AAAAA

The number 65 is the byte value for uppercase A.

Or you can fill the Buffer with a string:

const buffer = Buffer.alloc(5, "A", "utf8");

console.log(buffer.toString());

Output:

AAAAA

Use Buffer.alloc() when you need a known buffer size and want the memory initialized before writing data into it.

3. Buffer.allocUnsafe(): Allocate Without Initializing

Node.js also provides:

const buffer = Buffer.allocUnsafe(10);

The name is important.

allocUnsafe() can be faster because it does not guarantee that the allocated memory is initialized to zero. The Buffer may initially contain old data from a reused memory region.

That means you must completely initialize the relevant bytes before reading them or exposing them to another part of your application.

For example:

const buffer = Buffer.allocUnsafe(5);

buffer.fill(0);

console.log(buffer);

Output:

<Buffer 00 00 00 00 00>

We explicitly filled the allocated memory before using it.

For most beginner applications, Buffer.alloc() is the safer default. The performance trade-off of allocUnsafe() should only be considered when you understand the relevant memory behavior and can guarantee that the buffer is initialized correctly.

Never assume an unsafe buffer starts empty.

Part 5: Reading and Modifying Individual Bytes

One of the most useful properties of a Buffer is that you can access individual bytes by index.

import { Buffer } from "node:buffer";

const buffer = Buffer.from([10, 20, 30, 40]);

console.log(buffer[0]);
console.log(buffer[1]);
console.log(buffer[2]);

Output:

10
20
30

Like a normal JavaScript array, a Buffer uses zero-based indexing.

You can also modify an individual byte:

const buffer = Buffer.from([10, 20, 30]);

buffer[1] = 99;

console.log(buffer);

Output:

<Buffer 0a 63 1e>

The decimal value 99 appears as hexadecimal 63.

Each indexed Buffer value is a byte, so it represents a value in the range 0 through 255.

Let’s see what happens when we use a value outside that range:

const buffer = Buffer.alloc(1);

buffer[0] = 300;

console.log(buffer[0]);

Output:

44

For indexed assignment, the value is converted to an unsigned eight-bit value, so 300 wraps to 44, because 300 modulo 256 equals 44.

This behavior is useful to know when working with low-level data, but it can also hide bugs. When processing untrusted or externally supplied numbers, validate that the value is in the expected range before storing it.

Part 6: Converting Between Buffers and Strings

The most common conversion is turning encoded text into bytes and turning bytes back into text.

String to Buffer

const message = "Node.js";

const buffer = Buffer.from(message, "utf8");

console.log(buffer);

The string is encoded into a sequence of bytes.

Buffer to String

const message = buffer.toString("utf8");

console.log(message);

Output:

Node.js

The Buffer is decoded using UTF-8.

The encoding used during decoding should match the encoding used when the text was encoded. If it does not, the result may contain unexpected characters.

What If the Buffer Contains Invalid UTF-8?

Not every sequence of bytes is valid UTF-8.

When Buffer.toString("utf8") encounters invalid UTF-8 sequences, Node.js replaces invalid portions with the Unicode replacement character, �.

For example:

const buffer = Buffer.from([0x48, 0x69, 0xff]);

console.log(buffer.toString("utf8"));

The final byte FF is not a valid standalone UTF-8 byte, so the decoded string contains a replacement character for that invalid data.

This is another reason not to decode arbitrary binary files as text. The operation does not automatically determine the correct file format or encoding.

Part 7: Hexadecimal and Base64 Encoding

When inspecting binary data, you will often see hexadecimal or Base64 representations.

These are ways to represent bytes as text. They are not the same as the underlying data.

Hexadecimal

Hexadecimal uses the symbols 0 through 9 and A through F. Each pair of hexadecimal digits represents one byte.

For example:

const buffer = Buffer.from("Hi", "utf8");

console.log(buffer.toString("hex"));

Output:

4869

The string "4869" is a textual representation of the two bytes 48 and 69.

You can convert it back into a Buffer:

const restored = Buffer.from("4869", "hex");

console.log(restored.toString("utf8"));

Output:

Hi

Hexadecimal is especially useful for debugging network packets, inspecting file signatures, or examining low-level binary fields.

Base64

Base64 encodes binary data as printable ASCII characters. It is commonly used when binary content needs to be embedded in a text-based format.

const buffer = Buffer.from("Hello Node.js", "utf8");

const encoded = buffer.toString("base64");

console.log(encoded);

You can reverse the operation:

const decoded = Buffer.from(encoded, "base64");

console.log(decoded.toString("utf8"));

Output:

Hello Node.js

Base64 is an encoding, not encryption. Anyone with the encoded data can decode it without a secret key.

It also increases the amount of data required to represent the original bytes. Therefore, Base64 is useful for compatibility and transport, but it should not be treated as a security mechanism or automatically preferred for storing binary data.

Part 8: What Is the Difference Between Buffer and Uint8Array?

JavaScript provides typed arrays for working with numeric values in memory. One of these is Uint8Array, which represents unsigned eight-bit integers.

A Node.js Buffer is a specialized form of Uint8Array with additional methods for binary-data operations.

import { Buffer } from "node:buffer";

const buffer = Buffer.from([10, 20, 30]);

console.log(buffer instanceof Uint8Array);

Output:

true

This relationship is useful because Node.js can work with Buffers in APIs that accept suitable typed arrays.

The difference is that Buffer offers convenient operations such as:

buffer.toString("hex");
buffer.toString("base64");
buffer.write("Hello");

A plain Uint8Array does not provide the complete Buffer-specific API.

Both are useful for byte-oriented data, but Buffer is especially convenient for common Node.js file, stream, and network operations.

Part 9: Understanding Buffer Memory and subarray()

There is one important detail when working with Buffers: two Buffer objects can refer to the same underlying memory.

Consider this example:

const original = Buffer.from([10, 20, 30, 40]);

const part = original.subarray(1, 3);

console.log(part);

Output:

<Buffer 14 1e>

The new Buffer-like view contains the bytes at indexes 1 and 2 of the original.

Now modify the view:

part[0] = 99;

console.log(original);

Output:

<Buffer 0a 63 1e 28>

The original Buffer changed too.

Why?

Because subarray() creates a view over the same underlying memory rather than copying the selected bytes into an independent allocation.

This can be useful when processing large amounts of data without unnecessary copying, but it can also cause surprising changes when one part of an application modifies the data.

If you need an independent copy, use:

const original = Buffer.from([10, 20, 30, 40]);

const copy = Buffer.from(original);

copy[0] = 99;

console.log(original);
console.log(copy);

Output:

<Buffer 0a 14 1e 28>
<Buffer 63 14 1e 28>

Changing the copy does not modify the original Buffer.

Understanding the distinction between a view and a copy is important when working with binary data, especially when performance optimizations and shared memory are involved.

Part 10: Reading a File as Binary Data

Let’s reconnect Buffers to our previous lesson about the Node.js File System.

In the previous tutorial, we used code like this:

import * as fs from "node:fs/promises";

const text = await fs.readFile("notes.txt", "utf8");

Because we supplied "utf8", Node.js returned a string.

If we omit the encoding:

const data = await fs.readFile("notes.txt");

console.log(data);

Node.js returns a Buffer containing the file’s bytes.

This matters when dealing with a file that is not plain text.

Consider reading an image:

const imageData = await fs.readFile("photo.png");

console.log("Bytes:", imageData.length);
console.log("First bytes:", imageData.subarray(0, 8));

Many PNG files begin with this eight-byte signature:

89 50 4E 47 0D 0A 1A 0A

Those bytes help identify the file format. They are not ordinary readable text.

This example demonstrates a common binary-data workflow: read the bytes, inspect the relevant portion, and interpret the content according to the format specification.

Reading a whole file into a Buffer is appropriate for many small and medium-sized files. For very large files, use streams or chunked reads to avoid allocating memory for the entire file at once.

Part 11: Reading and Writing Numeric Values

Buffers are not just for text. They can also represent numeric fields in binary formats.

Suppose a binary message contains a 16-bit unsigned integer. A 16-bit value occupies two bytes.

There are two common byte orders to understand.

Big-endian stores the most significant byte first.

Little-endian stores the least significant byte first.

For the decimal value 258, the hexadecimal representation is 0x0102.

Therefore:

Big-endian:     01 02
Little-endian:  02 01

Node.js provides methods for reading and writing numbers in either byte order.

import { Buffer } from "node:buffer";

const buffer = Buffer.alloc(2);

buffer.writeUInt16BE(258, 0);

console.log(buffer);
console.log(buffer.readUInt16BE(0));

Output:

<Buffer 01 02>
258

The method name writeUInt16BE means “write an unsigned 16-bit integer in big-endian order.” The second argument is the byte offset at which to begin writing.

The matching readUInt16BE() method reads the number from that offset.

For little-endian encoding:

const buffer = Buffer.alloc(2);

buffer.writeUInt16LE(258, 0);

console.log(buffer);
console.log(buffer.readUInt16LE(0));

Output:

<Buffer 02 01>
258

The read and write methods must agree on byte order. If you write using big-endian but read using little-endian, the result will normally be a different value.

For larger integers, Node.js offers additional Buffer methods, including 32-bit and 64-bit variants. Always check that the Buffer is large enough for the specified read or write and that the value fits the numeric representation you choose.

Part 12: Practical Project: Build a Tiny Binary Message

Now let’s combine our knowledge of bytes, text encoding, and numeric fields into a small practical example.

Imagine that our application needs to create a binary message with this structure:

+------------------+------------------+--------------------+
| 1-byte type      | 2-byte data size  | UTF-8 payload       |
+------------------+------------------+--------------------+

The first byte identifies the message type. The next two bytes describe the payload length using big-endian order. The rest of the message contains the UTF-8-encoded text.

For example, the message "Hello" has a UTF-8 byte length of five. Its header might contain:

Type:          01
Payload size:  00 05
Payload:       48 65 6C 6C 6F

Let’s implement a function that creates the message:

import { Buffer } from "node:buffer";

function encodeMessage(type, message) {
  if (!Number.isInteger(type) || type < 0 || type > 255) {
    throw new RangeError("Message type must be a byte.");
  }

  const payload = Buffer.from(message, "utf8");

  if (payload.length > 65535) {
    throw new RangeError("Payload is too large.");
  }

  const header = Buffer.alloc(3);

  header.writeUInt8(type, 0);
  header.writeUInt16BE(payload.length, 1);

  return Buffer.concat([header, payload]);
}

const packet = encodeMessage(1, "Hello");

console.log(packet);
console.log(packet.toString("hex"));

The returned Buffer contains the header followed by the encoded payload.

Why is the payload length calculated using payload.length instead of message.length?

Because the length field describes the number of bytes in the binary message, not the number of JavaScript string code units. For a message containing non-ASCII characters, those measurements might be different.

Now decode the message:

function decodeMessage(packet) {
  if (packet.length < 3) {
    throw new Error("Message header is incomplete.");
  }

  const type = packet.readUInt8(0);
  const payloadLength = packet.readUInt16BE(1);

  if (packet.length !== 3 + payloadLength) {
    throw new Error("Payload length does not match the header.");
  }

  const payload = packet.subarray(3);

  return {
    type,
    message: payload.toString("utf8"),
  };
}

console.log(decodeMessage(packet));

Expected result:

{
  type: 1,
  message: 'Hello'
}

This example brings together several fundamental ideas.

The header contains numeric values, so we use numeric Buffer methods. The message contains text, so we encode it as UTF-8. The length field counts encoded bytes, not characters. Finally, the decoder validates the packet before attempting to interpret its payload.

This is the basic structure of many binary formats and protocols, although real-world formats may include version fields, checksums, encryption, compression, and more complex validation rules.

Part 13: Common Buffer Mistakes to Avoid

There are a few mistakes worth recognizing early.

Confusing string length with byte length. A string containing an emoji or characters from many writing systems may require more bytes than its length property suggests. Use Buffer.byteLength() or measure the encoded Buffer when a protocol requires byte length.

Decoding binary content as text without knowing its format. A PNG file, compressed archive, or encrypted payload should not be interpreted as UTF-8 text simply because it is stored in a Buffer.

Using Buffer.allocUnsafe() without initializing the data. Uninitialized memory may contain previous data. Only use this method if you can guarantee the relevant bytes are overwritten before they are read or exposed.

Assuming subarray() creates a copy. It creates a view over the original memory. Use Buffer.from(existingBuffer) when you need an independent copy.

Using Base64 as if it were encryption. Base64 changes how data is represented, not who can read it. Sensitive data still requires appropriate security measures.

Reading very large files into one Buffer. readFile() loads the entire file contents into memory. Streams or chunked reads are usually a better choice when processing large files.

Conclusion: Think in Bytes Before Thinking in Strings

Today, we explored a concept that underlies file handling, network communication, and binary data processing in Node.js.

We began with bits and bytes, learned how character encodings turn text into bytes, and saw why the number of characters in a JavaScript string can differ from the size of its UTF-8 representation.

From there, we explored the Buffer class, including how to create Buffers with from() and alloc(), inspect individual byte values, convert between Buffer and string representations, use hexadecimal and Base64, and distinguish a shared memory view from an independent copy.

Finally, we applied these concepts to reading files and constructing a simple binary message with a numeric header and a UTF-8 payload.

The most important takeaway is this: a Buffer represents bytes, not necessarily text. The meaning of those bytes depends on how your application interprets them. Once you understand that distinction, working with files, network packets, and binary formats becomes much easier.

Official Documentation and Further Reading

• Node.js Buffer Documentation
• Node.js File System Documentation
• Node.js Streams Documentation
• Node.js Learn: Reading Files