Unicode and font embedding

Every XFont has an encoding, and the encoding decides which characters the font can carry into the PDF. There are two: Unicode, the default, and WinAnsi, a single-byte encoding with room for 256 characters. This page explains the difference, how PdfPinata embeds the font file, and what happens to control characters in the strings you draw.

Choose an encoding

Pass XPdfFontOptions as the last argument of the XFont constructor. There is no property to change the encoding afterwards:

src/SampleApp/Demos/UnicodeDemo.cs
var winAnsi = new XFont("Liberation Sans", 13, XFontStyle.Regular,
    XPdfFontOptions.WinAnsiDefault);
var unicode = new XFont("Liberation Sans", 13, XFontStyle.Regular,
    XPdfFontOptions.UnicodeDefault);

	`XPdfFontOptions.UnicodeDefault`	`XPdfFontOptions.WinAnsiDefault`
Characters	Any character the font has a glyph for	The 256 characters of Windows code page 1252
Written as	A composite (CID) font that refers to glyphs by number	A simple font with one byte per character
Reordering, shaping and fallback	Yes	No
Copy and paste	Through a `/ToUnicode` map	Through the standard WinAnsi encoding

WinAnsi covers English and most Western European languages, but not Greek, Cyrillic, Polish, Czech, Turkish, arrows or most mathematical symbols. The demo draws the same strings in both encodings, side by side:

src/SampleApp/Demos/UnicodeDemo.cs
double y = 190;
foreach (var sample in samples)
{
    gfx1.DrawString(sample.Text, winAnsi, XBrushes.Black, new XPoint(50, y));
    gfx1.DrawString(sample.Text, unicode, XBrushes.Black, new XPoint(280, y));
    gfx1.DrawString(sample.What, body, XBrushes.DimGray, new XPoint(50, y + 13));
    y += 36;
}

Unicode is the right choice for almost every document. If you do not pass options, XFont uses GlobalFontSettings.DefaultFontEncoding, which is PdfFontEncoding.Unicode unless you change it.

PinataLayout uses WinAnsi unless you ask

PdfDocumentRenderer picks the encoding for every font in a PinataLayout document. Its parameterless constructor picks WinAnsi. Pass true to get Unicode:

var renderer = new PdfDocumentRenderer(true);   // Unicode encoding for all text

How fonts are embedded

PdfPinata always embeds the font file, whichever encoding you choose. What goes into the PDF depends on the kind of outlines the font has:

TrueType outlines, as in most .ttf files, are subset. Only the glyphs the document uses go into the file, so a document with one word in it carries a small part of the font.
PostScript (CFF) outlines, as in many .otf files, are embedded whole. A document that uses one character carries the same bytes as one that uses them all.

The second page of the demo saves five small documents and reads back the font object each one wrote. The embedded TrueType font holds only the few glyphs the word needs. The embedded CFF font is the whole file.

Chinese, Japanese and Korean text works with the Unicode encoding and a font that has the glyphs. CJK fonts are large, and CJK fonts with CFF outlines are embedded whole, so each document that uses one can grow by several megabytes. If a TrueType version of the font exists, use it.

Copying text back out

A reader can select, search and copy the text in a PdfPinata document. Each Unicode font carries a /ToUnicode map that says which characters each glyph stands for. The map covers glyphs from a shaper too, so a ligature copies out as the two or three letters it replaced. WinAnsi fonts use the standard encoding, which readers already understand.

To read text out of a PDF in your own code, see Text extraction.

Control characters

DrawString and MeasureString filter every string the same way before they use it:

A tab becomes a single space.
Every other character below 32 is dropped, including the line feed (\n) and the carriage return (\r).

DrawString always draws one line. A line feed does not start a new line; the words either side of it run together. It does not wrap long text either:

src/SampleApp/Demos/TextDemo.cs
gfx.DrawString("A newline\nvanishes between these words, a tab\tis the space it "
    + "measures as, and a long line runs off the edge of the page rather than "
    + "wrapping", body, XBrushes.Black,
    new XPoint(48, 510));

MeasureString treats a line feed differently: it splits the string there and reports the height of all the lines. For text with more than one line, use XTextFormatter, which breaks at line feeds and wraps to a width, or a PinataLayout paragraph.

Things to know

A character WinAnsi cannot hold is lost without an error. It is dropped or replaced when the document is written. You find out when somebody reads the document.
Set DefaultFontEncoding once, before the first font. Creating an XFont without options reads the setting, and reading it fixes it at Unicode. A later attempt to set a different value throws InvalidOperationException. Setting the same value again does nothing.
There is no way to turn embedding off. Every font the document uses is in the file. This is also what PDF/A requires.
Characters above U+FFFF work, for example many emoji, if the font has glyphs for them.
XPdfFontOptions takes only an encoding. It has no PdfFontEmbedding argument, because every font is embedded. See Migrating.

See it in action

The Unicode demo draws the same strings in both encodings, then saves small documents and reads back which kind of font object and which embedded font file each one wrote.

The full Unicode demo

src/SampleApp/Demos/UnicodeDemo.cs
var document = new PdfDocument();
document.Info.Title = "Unicode";

var heading = new XFont("Liberation Sans", 16, XFontStyle.Bold);
var label = new XFont("Liberation Sans", 9, XFontStyle.Bold);
var body = new XFont("Liberation Sans", 9);
var mono = new XFont("Source Code Pro", 8.5);

// The same family and size, differing only in what encoding the font is written with. The
// options go in the constructor: there is no property to change afterwards, because the
// encoding decides which kind of PDF font object gets built.
var winAnsi = new XFont("Liberation Sans", 13, XFontStyle.Regular,
    XPdfFontOptions.WinAnsiDefault);
var unicode = new XFont("Liberation Sans", 13, XFontStyle.Regular,
    XPdfFontOptions.UnicodeDefault);

(string Text, string What)[] samples =
{
    ("The quick brown fox", "Plain ASCII - both encodings carry it"),
    ("Café, naïve, Straße, £42", "Latin-1 - inside WinAnsi's 256 places"),
    ("Ελληνικά", "Greek - outside WinAnsi"),
    ("Кириллица", "Cyrillic - outside WinAnsi"),
    ("Ćwiczenia, Łódź", "Latin Extended - outside WinAnsi"),
    ("→ ← ↑ ↓ ∑ ∞", "Arrows and mathematics - outside WinAnsi")
};

// ----- page 1: what each encoding can carry -----

var page1 = document.AddPage();
var gfx1 = XGraphics.FromPdfPage(page1);
var prose1 = new XTextFormatter(gfx1);

gfx1.DrawString("What each encoding carries", heading, XBrushes.Black, new XPoint(50, 60));

prose1.DrawString(
    "An XFont carries an XPdfFontOptions, and the encoding in it decides what kind of font "
    + "object is written into the PDF. WinAnsi writes a simple font with a single-byte "
    + "encoding and 256 places to put a character in; Unicode writes a CID font, which has "
    + "no such limit. Both rows below are the same string in the same face at the same "
    + "size. Where they differ, the character was not one of WinAnsi's 256.",
    body, XBrushes.Black, new XRect(50, 80, 495, 60));

gfx1.DrawString("WinAnsiDefault", label, XBrushes.Black, new XPoint(50, 158));
gfx1.DrawString("UnicodeDefault", label, XBrushes.Black, new XPoint(280, 158));
gfx1.DrawLine(new XPen(XColors.Gainsboro, 0.5), 50, 165, 545, 165);

double y = 190;
foreach (var sample in samples)
{
    gfx1.DrawString(sample.Text, winAnsi, XBrushes.Black, new XPoint(50, y));
    gfx1.DrawString(sample.Text, unicode, XBrushes.Black, new XPoint(280, y));
    gfx1.DrawString(sample.What, body, XBrushes.DimGray, new XPoint(50, y + 13));
    y += 36;
}

prose1.DrawString(
    "Nothing threw. A character WinAnsi has no place for is not an error - it is dropped or "
    + "replaced on the way out, which is exactly the failure that gets noticed after the "
    + "document has been sent. Unicode is the safe default and is what "
    + "PdfDocumentRenderer(unicode: true) selects for a PinataLayout document, under a name "
    + "that gives no hint that it is this setting.",
    body, XBrushes.Black, new XRect(50, y + 10, 495, 60));

// ----- page 2: what it does to the file -----

// Two one-string documents, saved, so the difference can be read out of the files rather
// than described. Nothing is written to disk.
(string Encoding, string Subtype, string FontFile, int Length, long Bytes) Probe(
    XPdfFontOptions options, string text, string family)
{
    using var buffer = new MemoryStream();
    using (var probe = new PdfDocument())
    {
        probe.Options.CompressContentStreams = true;
        var page = probe.AddPage();
        using (var gfx = XGraphics.FromPdfPage(page))
        {
            gfx.DrawString(text, new XFont(family, 12, XFontStyle.Regular, options),
                XBrushes.Black, new XPoint(50, 50));
        }

        probe.Save(buffer, false);
    }

    buffer.Position = 0;
    using var reopened = PdfReader.Open(buffer, PdfDocumentOpenMode.Import);

    // Walk the page's font resources and report what kind of font object was written and
    // which key the face's bytes ended up under.
    var fonts = reopened.Pages[0].Elements
        .GetDictionary("/Resources")?.Elements.GetDictionary("/Font");

    string subtype = "none", fontFile = "none";
    var length = 0;

    if (fonts != null)
    {
        foreach (var key in fonts.Elements.KeyNames.Select(name => name.Value))
        {
            var font = fonts.Elements.GetDictionary(key);
            subtype = font.Elements.GetName("/Subtype");

            // A CID font hides the descriptor one level down, under /DescendantFonts.
            var descriptor = font.Elements.GetDictionary("/FontDescriptor");
            if (descriptor == null)
            {
                var descendants = font.Elements.GetArray("/DescendantFonts");
                if (descendants != null && descendants.Elements.Count > 0)
                {
                    descriptor = (descendants.Elements.GetDictionary(0))
                        ?.Elements.GetDictionary("/FontDescriptor");
                }
            }

            if (descriptor != null)
            {
                foreach (var file in new[] { "/FontFile", "/FontFile2", "/FontFile3" })
                {
                    var embedded = descriptor.Elements.GetDictionary(file);
                    if (embedded != null)
                    {
                        fontFile = file;
                        length = embedded.Stream?.Length ?? 0;
                    }
                }
            }
        }
    }

    return (options.FontEncoding.ToString(), subtype, fontFile, length, buffer.Length);
}

var probes = new[]
{
    Probe(XPdfFontOptions.WinAnsiDefault, "Hello", "Liberation Sans"),
    Probe(XPdfFontOptions.UnicodeDefault, "Hello", "Liberation Sans"),
    Probe(XPdfFontOptions.UnicodeDefault, "Кириллица", "Liberation Sans"),
    Probe(XPdfFontOptions.WinAnsiDefault, "Hello", "Source Code Pro"),
    Probe(XPdfFontOptions.UnicodeDefault, "Hello", "Source Code Pro")
};

string[] descriptions =
{
    "Liberation Sans, WinAnsi", "Liberation Sans, Unicode",
    "Liberation Sans, Unicode, Cyrillic", "Source Code Pro, WinAnsi",
    "Source Code Pro, Unicode"
};

var page2 = document.AddPage();
var gfx2 = XGraphics.FromPdfPage(page2);
var prose2 = new XTextFormatter(gfx2);

gfx2.DrawString("What it does to the file", heading, XBrushes.Black, new XPoint(50, 60));

prose2.DrawString(
    "Five one-word documents, saved and reopened, with the font object each of them wrote "
    + "read back out. The subtype is the kind of PDF font; the key is where the face's own "
    + "bytes ended up.",
    body, XBrushes.Black, new XRect(50, 80, 495, 40));

gfx2.DrawString("document", label, XBrushes.Black, new XPoint(50, 135));
gfx2.DrawString("subtype", label, XBrushes.Black, new XPoint(230, 135));
gfx2.DrawString("key", label, XBrushes.Black, new XPoint(340, 135));
gfx2.DrawString("face bytes", label, XBrushes.Black, new XPoint(420, 135));
gfx2.DrawString("file", label, XBrushes.Black, new XPoint(490, 135));

double row = 155;
for (var index = 0; index < probes.Length; index++)
{
    gfx2.DrawString(descriptions[index], body, XBrushes.Black, new XPoint(50, row));
    gfx2.DrawString(probes[index].Subtype, mono, XBrushes.Firebrick, new XPoint(230, row));
    gfx2.DrawString(probes[index].FontFile, mono, XBrushes.Firebrick, new XPoint(340, row));
    gfx2.DrawString($"{probes[index].Length:N0}", body, XBrushes.Black, new XPoint(420, row));
    gfx2.DrawString($"{probes[index].Bytes:N0}", body, XBrushes.DimGray, new XPoint(490, row));
    row += 16;
}

gfx2.DrawString("Two outlines, two embedding paths", label, XBrushes.Black,
    new XPoint(50, row + 20));

prose2.DrawString(
    "Liberation Sans is TrueType, so only the glyphs the document uses are embedded - the "
    + "face is subsetted, and the two Liberation rows differ in size for that reason "
    + "alone. Source Code Pro has PostScript (CFF) outlines, which this library embeds "
    + "whole under /FontFile3 because a CFF subsetter is not written; a document using one "
    + "character of it carries the same bytes as a document using all of them. That is the "
    + "trade recorded in font-embedding-gaps.md, and it is why the size column moves for "
    + "one face and not for the other.",
    body, XBrushes.Black, new XRect(50, row + 33, 495, 80));

gfx2.DrawString("Why CJK is not on this page", label, XBrushes.Black, new XPoint(50, row + 125));

prose2.DrawString(
    "Nothing here is a limit of the library: a CID font carries any character a face has a "
    + "glyph for, and this app simply does not carry a face that has CJK glyphs. Liberation "
    + "Sans covers Latin, Latin Extended, Greek and Cyrillic, which is what page one uses. "
    + "A CJK face is several megabytes for one panel, so the demo shows the mechanism and "
    + "leaves the asset to whoever needs it - register any face through IFontResolver and "
    + "the Unicode path above carries it.",
    body, XBrushes.Black, new XRect(50, row + 138, 495, 80));

Choose an encoding​

PinataLayout uses WinAnsi unless you ask​

How fonts are embedded​

Copying text back out​

Control characters​

Things to know​

See it in action​