@portable/charset

`@portable/charset`

Functions

`Charset.encode`

Encodes text into bytes in the given charset. Bytes are read as UTF-8 text, and a UTF-8 byte order mark at their start is dropped.

Lone surrogates are encoded as U+FFFD, as TextEncoder does.

function encode(
  text: string | Uint8Array,
  charset: Charset,
  options: encode.Options = {},
): Uint8Array<ArrayBuffer>;

Option	Type	Default	Description
`bom`	`boolean`	`false`	Write a byte order mark first. Applies only to UTF-8, UTF-16LE and UTF-16BE, because other charsets have none.
`unmappable`	`Unmappable`	`'replace'`	What to write for a character the charset can't represent: `'replace'` writes `?`, `'html'` writes an HTML numeric character reference such as `日`, `'error'` throws `UnsupportedError`, and a function returns the text to write instead.

import * as Charset from '@portable/charset';
import { windows1251 } from '@portable/charset/single-byte';
 
Charset.encode('Привет', windows1251); // Uint8Array [0xcf, 0xf0, 0xe8, 0xe2, 0xe5, 0xf2]
Charset.encode('日本', windows1251, { unmappable: 'html' }); // the bytes of '&#26085;&#26412;'

`Charset.normalizeLabel`

Returns the WHATWG Encoding Standard name for a charset label, or undefined for an unknown label. Accepts every WHATWG label and every iconv-lite name for the same charsets. Leading and trailing ASCII whitespace, case, punctuation and a trailing year such as :1988 are ignored. A label with any other character, such as NUL or a no-break space, returns undefined.

Where iconv-lite disagrees with WHATWG, the WHATWG mapping is used: latin1, ascii and iso-8859-1 are windows-1252, and gb2312 is GBK. Labels mapped to the replacement encoding, such as iso-2022-kr, return undefined, because the replacement encoding has no encoder. This package can encode a label exactly when normalizeLabel returns a name.

function normalizeLabel(label: string): Name | undefined;

import * as Charset from '@portable/charset';
 
Charset.normalizeLabel('sjis'); // 'Shift_JIS'
Charset.normalizeLabel(' Latin1 '); // 'windows-1252'
Charset.normalizeLabel('ISO_8859-5:1988'); // 'ISO-8859-5'
Charset.normalizeLabel('cp437'); // undefined
Charset.normalizeLabel('iso-2022-kr'); // undefined

`Charset.load`

Loads the charset for a label, for when the charset is only known at runtime, such as from a Content-Type header. Each family's tables are imported on first use, so bundlers split them into chunks that load only when needed. Throws UnsupportedError for a label normalizeLabel doesn't know.

async function load(
  label: string,
  options: load.Options = {},
): Promise<Charset>;

Option	Type	Default	Description
`signal`	`AbortSignal`		Rejects with `signal.reason` when aborted. The import itself keeps running.

import * as Charset from '@portable/charset';
 
Charset.encode('日本', await Charset.load('windows-31j')); // Uint8Array [0x93, 0xfa, 0x96, 0x7b]

Classes

`Charset.EncoderStream`

A TransformStream that encodes text into bytes in the given charset, like TextEncoderStream for legacy charsets. Byte chunks are read as UTF-8. A character split across chunks is encoded once, and ISO-2022-JP's escape back to ASCII is written when the stream closes.

class EncoderStream extends TransformStream<
  string | Uint8Array,
  Uint8Array<ArrayBuffer>
> {
  constructor(charset: Charset, options: EncoderStream.Options = {});
}

Option	Type	Default	Description
`bom`	`boolean`	`false`	Write a byte order mark first. Applies only to UTF-8, UTF-16LE and UTF-16BE, because other charsets have none.
`unmappable`	`import("./internal/run.js").Unmappable`	`'replace'`	What to write for a character the charset can't represent: `'replace'` writes `?`, `'html'` writes an HTML numeric character reference such as `日`, `'error'` throws `UnsupportedError`, and a function returns the text to write instead.
`signal`	`AbortSignal`		Errors the stream with `signal.reason` when the signal aborts, which makes `pipeThrough` cancel the source.

import * as Charset from '@portable/charset';
import { shiftJis } from '@portable/charset/shift_jis';
 
const response = await fetch('https://example.com/report.csv');
const shiftJisBytes = response.body!.pipeThrough(
  new Charset.EncoderStream(shiftJis),
);

Constants

`Charset.utf8`

UTF-8, encoded with the platform's TextEncoder. Use it when the charset comes from a label at runtime and may be UTF-8. Otherwise TextEncoder does the same job.

const utf8: Charset;

import * as Charset from '@portable/charset';
 
Charset.encode('é', Charset.utf8, { bom: true }); // Uint8Array [0xef, 0xbb, 0xbf, 0xc3, 0xa9]

Errors

`Charset.UnsupportedError`

Thrown by load when it doesn't know the label or can't encode its charset, and by encode and EncoderStream when unmappable is 'error' and the text has a character the charset can't represent.

class UnsupportedError extends Errors.UnsupportedError {
  readonly kind: string;
}

import * as Charset from '@portable/charset';
import { windows1251 } from '@portable/charset/single-byte';
 
try {
  Charset.encode('日本', windows1251, { unmappable: 'error' });
} catch (error) {
  error instanceof Charset.UnsupportedError; // true
}

Types

`Charset.Charset`

A charset that encode and EncoderStream can write. Import one from the subpath for its family, such as shiftJis from @portable/charset/shift_jis, or get one by label with load.

type Charset = {
  readonly name: Name;
  readonly encode: (text: string) => Uint8Array<ArrayBuffer>;
  readonly [definition]: true;
};

`Charset.Name`

The WHATWG Encoding Standard name of a charset this package encodes, such as 'Shift_JIS'.

type Name = Exclude<CharsetName>;

`@portable/charset/single-byte`

Constants

`Charset.ibm866`

IBM866.

const ibm866: Charset;

import * as Charset from '@portable/charset';
import { ibm866 } from '@portable/charset/single-byte';
 
Charset.encode('р', ibm866); // Uint8Array [0xe0]

`Charset.iso8859_2`

ISO-8859-2.

const iso8859_2: Charset;

import * as Charset from '@portable/charset';
import { iso8859_2 } from '@portable/charset/single-byte';
 
Charset.encode('ŕ', iso8859_2); // Uint8Array [0xe0]

`Charset.iso8859_3`

ISO-8859-3.

const iso8859_3: Charset;

import * as Charset from '@portable/charset';
import { iso8859_3 } from '@portable/charset/single-byte';
 
Charset.encode('à', iso8859_3); // Uint8Array [0xe0]

`Charset.iso8859_4`

ISO-8859-4.

const iso8859_4: Charset;

import * as Charset from '@portable/charset';
import { iso8859_4 } from '@portable/charset/single-byte';
 
Charset.encode('ā', iso8859_4); // Uint8Array [0xe0]

`Charset.iso8859_5`

ISO-8859-5.

const iso8859_5: Charset;

import * as Charset from '@portable/charset';
import { iso8859_5 } from '@portable/charset/single-byte';
 
Charset.encode('р', iso8859_5); // Uint8Array [0xe0]

`Charset.iso8859_6`

ISO-8859-6.

const iso8859_6: Charset;

import * as Charset from '@portable/charset';
import { iso8859_6 } from '@portable/charset/single-byte';
 
Charset.encode('ـ', iso8859_6); // Uint8Array [0xe0]

`Charset.iso8859_7`

ISO-8859-7.

const iso8859_7: Charset;

import * as Charset from '@portable/charset';
import { iso8859_7 } from '@portable/charset/single-byte';
 
Charset.encode('ΰ', iso8859_7); // Uint8Array [0xe0]

`Charset.iso8859_8`

ISO-8859-8.

const iso8859_8: Charset;

import * as Charset from '@portable/charset';
import { iso8859_8 } from '@portable/charset/single-byte';
 
Charset.encode('א', iso8859_8); // Uint8Array [0xe0]

`Charset.iso8859_8i`

ISO-8859-8-I. The same bytes as ISO-8859-8, but the name tells a reader the text is in logical order.

const iso8859_8i: Charset;

import * as Charset from '@portable/charset';
import { iso8859_8i } from '@portable/charset/single-byte';
 
Charset.encode('א', iso8859_8i); // Uint8Array [0xe0]

`Charset.iso8859_10`

ISO-8859-10.

const iso8859_10: Charset;

import * as Charset from '@portable/charset';
import { iso8859_10 } from '@portable/charset/single-byte';
 
Charset.encode('ā', iso8859_10); // Uint8Array [0xe0]

`Charset.iso8859_13`

ISO-8859-13.

const iso8859_13: Charset;

import * as Charset from '@portable/charset';
import { iso8859_13 } from '@portable/charset/single-byte';
 
Charset.encode('ą', iso8859_13); // Uint8Array [0xe0]

`Charset.iso8859_14`

ISO-8859-14.

const iso8859_14: Charset;

import * as Charset from '@portable/charset';
import { iso8859_14 } from '@portable/charset/single-byte';
 
Charset.encode('à', iso8859_14); // Uint8Array [0xe0]

`Charset.iso8859_15`

ISO-8859-15.

const iso8859_15: Charset;

import * as Charset from '@portable/charset';
import { iso8859_15 } from '@portable/charset/single-byte';
 
Charset.encode('à', iso8859_15); // Uint8Array [0xe0]

`Charset.iso8859_16`

ISO-8859-16.

const iso8859_16: Charset;

import * as Charset from '@portable/charset';
import { iso8859_16 } from '@portable/charset/single-byte';
 
Charset.encode('à', iso8859_16); // Uint8Array [0xe0]

`Charset.koi8R`

KOI8-R.

const koi8R: Charset;

import * as Charset from '@portable/charset';
import { koi8R } from '@portable/charset/single-byte';
 
Charset.encode('Ю', koi8R); // Uint8Array [0xe0]

`Charset.koi8U`

KOI8-U.

const koi8U: Charset;

import * as Charset from '@portable/charset';
import { koi8U } from '@portable/charset/single-byte';
 
Charset.encode('Ю', koi8U); // Uint8Array [0xe0]

`Charset.macintosh`

macintosh.

const macintosh: Charset;

import * as Charset from '@portable/charset';
import { macintosh } from '@portable/charset/single-byte';
 
Charset.encode('‡', macintosh); // Uint8Array [0xe0]

`Charset.windows874`

windows-874.

const windows874: Charset;

import * as Charset from '@portable/charset';
import { windows874 } from '@portable/charset/single-byte';
 
Charset.encode('เ', windows874); // Uint8Array [0xe0]

`Charset.windows1250`

windows-1250.

const windows1250: Charset;

import * as Charset from '@portable/charset';
import { windows1250 } from '@portable/charset/single-byte';
 
Charset.encode('ŕ', windows1250); // Uint8Array [0xe0]

`Charset.windows1251`

windows-1251.

const windows1251: Charset;

import * as Charset from '@portable/charset';
import { windows1251 } from '@portable/charset/single-byte';
 
Charset.encode('а', windows1251); // Uint8Array [0xe0]

`Charset.windows1252`

windows-1252.

const windows1252: Charset;

import * as Charset from '@portable/charset';
import { windows1252 } from '@portable/charset/single-byte';
 
Charset.encode('à', windows1252); // Uint8Array [0xe0]

`Charset.windows1253`

windows-1253.

const windows1253: Charset;

import * as Charset from '@portable/charset';
import { windows1253 } from '@portable/charset/single-byte';
 
Charset.encode('ΰ', windows1253); // Uint8Array [0xe0]

`Charset.windows1254`

windows-1254.

const windows1254: Charset;

import * as Charset from '@portable/charset';
import { windows1254 } from '@portable/charset/single-byte';
 
Charset.encode('à', windows1254); // Uint8Array [0xe0]

`Charset.windows1255`

windows-1255.

const windows1255: Charset;

import * as Charset from '@portable/charset';
import { windows1255 } from '@portable/charset/single-byte';
 
Charset.encode('א', windows1255); // Uint8Array [0xe0]

`Charset.windows1256`

windows-1256.

const windows1256: Charset;

import * as Charset from '@portable/charset';
import { windows1256 } from '@portable/charset/single-byte';
 
Charset.encode('à', windows1256); // Uint8Array [0xe0]

`Charset.windows1257`

windows-1257.

const windows1257: Charset;

import * as Charset from '@portable/charset';
import { windows1257 } from '@portable/charset/single-byte';
 
Charset.encode('ą', windows1257); // Uint8Array [0xe0]

`Charset.windows1258`

windows-1258.

const windows1258: Charset;

import * as Charset from '@portable/charset';
import { windows1258 } from '@portable/charset/single-byte';
 
Charset.encode('à', windows1258); // Uint8Array [0xe0]

`Charset.xMacCyrillic`

x-mac-cyrillic.

const xMacCyrillic: Charset;

import * as Charset from '@portable/charset';
import { xMacCyrillic } from '@portable/charset/single-byte';
 
Charset.encode('а', xMacCyrillic); // Uint8Array [0xe0]

`Charset.xUserDefined`

x-user-defined, which maps bytes 0x80 to 0xFF to the private use code points U+F780 to U+F7FF.

const xUserDefined: Charset;

import * as Charset from '@portable/charset';
import { xUserDefined } from '@portable/charset/single-byte';
 
Charset.encode('\uf7ff', xUserDefined); // Uint8Array [0xff]

`@portable/charset/shift_jis`

Constants

`Charset.shiftJis`

Shift_JIS as the WHATWG Encoding Standard defines it, which is Windows code page 932 (windows-31j, cp932) including the NEC and IBM extensions.

const shiftJis: Charset;

import * as Charset from '@portable/charset';
import { shiftJis } from '@portable/charset/shift_jis';
 
Charset.encode('日本ｶﾅ', shiftJis); // Uint8Array [0x93, 0xfa, 0x96, 0x7b, 0xb6, 0xc5]

`@portable/charset/euc-jp`

Constants

`Charset.eucJp`

EUC-JP. As the WHATWG Encoding Standard specifies, it encodes JIS X 0208 and halfwidth katakana only. WHATWG decoders read JIS X 0212 characters, but eucJp never encodes them.

const eucJp: Charset;

import * as Charset from '@portable/charset';
import { eucJp } from '@portable/charset/euc-jp';
 
Charset.encode('日本', eucJp); // Uint8Array [0xc6, 0xfc, 0xcb, 0xdc]

`@portable/charset/iso-2022-jp`

Constants

`Charset.iso2022Jp`

ISO-2022-JP, the 7-bit Japanese charset used in email. It switches between ASCII, JIS X 0201 Roman and JIS X 0208 with escape sequences and always ends in ASCII. Halfwidth katakana are encoded as their fullwidth forms.

const iso2022Jp: Charset;

import * as Charset from '@portable/charset';
import { iso2022Jp } from '@portable/charset/iso-2022-jp';
 
Charset.encode('a日', iso2022Jp); // Uint8Array [0x61, 0x1b, 0x24, 0x42, 0x46, 0x7c, 0x1b, 0x28, 0x42]

`@portable/charset/gbk`

Constants

`Charset.gbk`

GBK, which the WHATWG Encoding Standard also uses for the labels gb2312 and x-gbk. It is gb18030 without the four-byte sequences, and it writes € as the single byte 0x80.

const gbk: Charset;

import * as Charset from '@portable/charset';
import { gbk } from '@portable/charset/gbk';
 
Charset.encode('中国€', gbk); // Uint8Array [0xd6, 0xd0, 0xb9, 0xfa, 0x80]

`Charset.gb18030`

gb18030, which encodes every Unicode scalar value except U+E5E5.

const gb18030: Charset;

import * as Charset from '@portable/charset';
import { gb18030 } from '@portable/charset/gbk';
 
Charset.encode('中€😀', gb18030); // Uint8Array [0xd6, 0xd0, 0xa2, 0xe3, 0x94, 0x39, 0xfc, 0x36]

`@portable/charset/big5`

Constants

`Charset.big5`

Big5 as the WHATWG Encoding Standard defines it, which decodes the Hong Kong Supplementary Character Set but never encodes it, so plain Big5 decoders can read the output.

const big5: Charset;

import * as Charset from '@portable/charset';
import { big5 } from '@portable/charset/big5';
 
Charset.encode('中文', big5); // Uint8Array [0xa4, 0xa4, 0xa4, 0xe5]

`@portable/charset/euc-kr`

Constants

`Charset.eucKr`

EUC-KR as the WHATWG Encoding Standard defines it, which is Windows code page 949 (Unified Hangul Code) and covers every Hangul syllable.

const eucKr: Charset;

import * as Charset from '@portable/charset';
import { eucKr } from '@portable/charset/euc-kr';
 
Charset.encode('한국', eucKr); // Uint8Array [0xc7, 0xd1, 0xb1, 0xb9]

`@portable/charset/utf-16`

Constants

`Charset.utf16le`

UTF-16 little-endian. The WHATWG labels utf-16 and unicode map to this charset. Pass bom: true to encode to write the byte order mark iconv-lite writes for utf-16.

const utf16le: Charset;

import * as Charset from '@portable/charset';
import { utf16le } from '@portable/charset/utf-16';
 
Charset.encode('a€', utf16le); // Uint8Array [0x61, 0x00, 0xac, 0x20]

`Charset.utf16be`

UTF-16 big-endian.

const utf16be: Charset;

import * as Charset from '@portable/charset';
import { utf16be } from '@portable/charset/utf-16';
 
Charset.encode('a€', utf16be); // Uint8Array [0x00, 0x61, 0x20, 0xac]