Reading content streams

Everything drawn on a PDF page is stored in the page's content stream: a list of operators, each with its operands. 100 200 50 30 re adds a rectangle to the path, S strokes it, and (Hello) Tj shows text. ContentReader parses that list into objects you can read in C#.

Read a page's content when you need to know what is really on it:

• to find out why a page is blank. A page whose operators are all there but whose coordinates are wrong looks the same as a page with nothing on it, and only the content tells you which it is;
• to check in a test what your code drew, without rendering the page;
• to see what a page from another producer is made of: mostly text, mostly paths, or one large image;
• to make a simple change, such as removing an operator, and write the content back.

ContentReader is in PdfPinata.Pdf.Content, and the objects it returns are in PdfPinata.Pdf.Content.Objects. Both are in the core PdfPinata package. To read the text of a page rather than its operators, use Text extraction instead.

Read a page's operators

ContentReader.ReadContent(page) reads every content stream of the page, decompresses it and parses it. The Inspect demo saves the document it has drawn and opens it again, so that it reads the content exactly as it was written to the file:

src/SampleApp/Demos/InspectDemo.cs

// The page has to be saved and reopened before its content can be read: what is being
// read is the content stream as it was written, and until the document is saved there is
// no stream to read. This is the same round trip the test suite's helpers make.
byte[] saved;
using (var buffer = new MemoryStream())
{
    document.Save(buffer, false);
    saved = buffer.ToArray();
}

var operators = new List<COperator>();
using (var buffer = new MemoryStream(saved))
{
    using var reopened = PdfReader.Open(buffer, PdfDocumentOpenMode.Import);

    // One call. What comes back is a CSequence - a list of CObject, most of which are the
    // operators, each carrying the operands it was given.
    var content = ContentReader.ReadContent(reopened.Pages[0]);
    operators.AddRange(content.OfType<COperator>());
}

ReadContent also takes a byte[] or a MemoryStream of content that has already been decompressed, for example the stream of a form XObject.

The object model

ReadContent returns a CSequence: a list of CObject. You can index it, loop over it with foreach, and query it with LINQ. Each operator is a COperator, and its operands are a CSequence of their own:

Type	Holds
COperator	An operator. OpCode.Name is its name as the file writes it, such as "re" or "Tj". OpCode.OpCodeName is the same as an enum value. OpCode.Description explains it. Operands holds its operands.
CInteger, CReal	A number, in Value.
CName	A name, such as a font's resource name /F0, in Name.
CString	A string, in Value, one character per byte.
CArray	An array of operands. It is a CSequence, so you can index and count it.
CComment	A comment in the content, in Text.

The Inspect demo prints each operand by its type:

src/SampleApp/Demos/InspectDemo.cs

// The operand types are the whole of the CObject model: numbers, strings, names and
// arrays. Rendering them by type is what makes the model visible rather than the text.
string Describe(CObject operand) => operand switch
{
    CInteger integer => integer.Value.ToString(),
    CReal real => real.Value.ToString("0.###"),
    CName name => name.Name,
    CString text => $"({text.Value.Length} bytes)",
    CArray array => $"[{array.Count} items]",
    _ => operand.ToString() ?? ""
};

Count what a page is made of

A count by operator is often more useful than the full list. It shows at a glance whether a page is mostly text, paths or images:

src/SampleApp/Demos/InspectDemo.cs

var counted = operators
    .GroupBy(op => op.OpCode.Name)
    .Select(group => new { Name = group.Key, Count = group.Count() })
    .OrderByDescending(entry => entry.Count)
    .ThenBy(entry => entry.Name)
    .ToList();

These operators turn up most often:

Operator	Means
q, Q	Save and restore the graphics state.
cm	Change the coordinate system.
re, m, l, c, h	Build a path: rectangle, move to, line to, curve to, close.
S, f, B, n	Stroke the path, fill it, do both, or do neither.
W	Use the path as a clipping path.
rg, RG	Set the fill colour and the stroke colour.
w	Set the line width.
BT, ET	Begin and end a text object.
Tf	Set the font and size.
Td	Move to the start of the next line of text.
Tj, TJ	Show text. TJ takes an array with spacing between the parts.
gs	Apply a set of graphics state parameters, such as transparency.
Do	Paint an image or a form XObject.

The operands of Tf, gs and Do are resource names. The page's resource dictionary says which font, graphics state or image each name stands for. See Working with PDF objects.

Why the text reads as numbers

In a document written by PdfPinata, the operands of Tj are not readable words. Fonts are embedded as Unicode by default, and a Unicode font's text is written as two-byte glyph numbers, which mean nothing without the font. Other producers do the same. To get the characters, use PdfTextExtractor, which translates glyph numbers through the font's Unicode map.

If you set XPdfFontOptions.WinAnsiDefault on a font, its text is written as readable strings instead, but only characters in the WinAnsi character set can be drawn. See Unicode and font embedding.

Change the content and write it back

To change a page's content, change the sequence and pass it to page.Contents.ReplaceContent. The document must be open in Modify or Append mode. This removes every image and form that a page paints:

using PdfPinata.Pdf.Content;
using PdfPinata.Pdf.Content.Objects;

CSequence content = ContentReader.ReadContent(page);
for (int index = content.Count - 1; index >= 0; index--)
{
    if (content[index] is COperator op && op.OpCode.Name == "Do")
        content.RemoveAt(index);
}
page.Contents.ReplaceContent(content);
document.PruneUnusedResources();

ReplaceContent writes the sequence as the page's single content stream. The images themselves are still named in the page's resources until PruneUnusedResources removes them.

To build a new operator, call OpCodes.OperatorFromName("re") and add its operands to Operands.

Things to know

• Read the saved content. The Inspect demo saves and reopens its document before it reads a page it has drawn in the same run. Reading a page from a file you opened needs no such step.
• An inline image is one object. A small image can be written directly into the content, between the BI and EI operators. The reader gives it to you as a CInlineImage, a COperator named BI. Its ImageDictionary holds the entries as written, and its Data holds the image bytes. Content that you write back keeps the image. The reader finds the end of the data by looking for the bytes EI, and binary data can contain these bytes. If it does, the reader stops too early.
• Content that does not parse throws ContentReaderException.
• Reading a page's content can change how the page stores it. ReadContent(page) gathers the page's content streams into an array on the page. The page draws the same, but in a document open in Append mode the page counts as changed and is written into the next revision.
• Only the page's own content is read. A Do operator paints a form XObject whose content is a stream of its own. Read that stream separately if you need it.

See it in action

The Inspect demo draws a page with six calls, reads the page back, lists the operators the calls produced, and counts them.

The full Inspect demo

src/SampleApp/Demos/InspectDemo.cs

var document = new PdfDocument();
document.Info.Title = "Inspect";

var heading = new XFont("Liberation Sans", 16, XFontStyle.Bold);
var label = new XFont("Liberation Sans", 9, XFontStyle.Bold);
var body = new XFont("Liberation Sans", 9);
var mono = new XFont("Source Code Pro", 7.5);

// ----- page 1: something worth reading back -----

var subject = document.AddPage();
using (var gfx = XGraphics.FromPdfPage(subject))
{
    gfx.DrawString("The page being read", heading, XBrushes.Black, new XPoint(50, 60));

    new XTextFormatter(gfx).DrawString(
        "A deliberately short page, so that the operators it produces fit on the next one "
        + "and can be read against the calls that made them. Six calls: a string, a "
        + "rectangle with a pen and a brush, a line, an ellipse, a path and a second "
        + "string in another colour.",
        body, XBrushes.Black, new XRect(50, 80, 495, 50));

    gfx.DrawRectangle(new XPen(XColors.MidnightBlue, 2),
        new XSolidBrush(XColor.FromArgb(60, 70, 130, 180)), 50, 150, 200, 100);
    gfx.DrawLine(new XPen(XColors.Firebrick, 3), 300, 150, 500, 250);
    gfx.DrawEllipse(new XPen(XColors.SeaGreen, 1.5), null, 50, 280, 200, 100);

    var path = new XGraphicsPath();
    path.AddPolygon(new[]
    {
        new XPoint(320, 290), new XPoint(420, 290), new XPoint(370, 370)
    });
    gfx.DrawPath(new XPen(XColors.DarkOrange, 1.5), path);

    gfx.DrawString("Six calls, and the operators overleaf", body, XBrushes.Firebrick,
        new XPoint(50, 410));
}

// The page has to be saved and reopened before its content can be read: what is being
// read is the content stream as it was written, and until the document is saved there is
// no stream to read. This is the same round trip the test suite's helpers make.
byte[] saved;
using (var buffer = new MemoryStream())
{
    document.Save(buffer, false);
    saved = buffer.ToArray();
}

var operators = new List<COperator>();
using (var buffer = new MemoryStream(saved))
{
    using var reopened = PdfReader.Open(buffer, PdfDocumentOpenMode.Import);

    // One call. What comes back is a CSequence - a list of CObject, most of which are the
    // operators, each carrying the operands it was given.
    var content = ContentReader.ReadContent(reopened.Pages[0]);
    operators.AddRange(content.OfType<COperator>());
}

// ----- page 2: the operators themselves -----

var listing = document.AddPage();
using (var gfx = XGraphics.FromPdfPage(listing))
{
    var prose = new XTextFormatter(gfx);

    gfx.DrawString("What the page is made of", heading, XBrushes.Black, new XPoint(50, 60));

    prose.DrawString(
        $"The previous page came back as {operators.Count} operators. The first sixty are "
        + "below, each with its operands, in the order they were written. An operator's "
        + "name is the PDF one rather than the drawing call's - re for a rectangle, S to "
        + "stroke, f to fill, B to do both, Tj to show text.",
        body, XBrushes.Black, new XRect(50, 80, 495, 50));

    // The operand types are the whole of the CObject model: numbers, strings, names and
    // arrays. Rendering them by type is what makes the model visible rather than the text.
    string Describe(CObject operand) => operand switch
    {
        CInteger integer => integer.Value.ToString(),
        CReal real => real.Value.ToString("0.###"),
        CName name => name.Name,
        CString text => $"({text.Value.Length} bytes)",
        CArray array => $"[{array.Count} items]",
        _ => operand.ToString() ?? ""
    };

    double y = 140;
    double x = 50;
    foreach (var op in operators.Take(60))
    {
        var operands = string.Join(" ", op.Operands.Select(Describe));
        gfx.DrawString(op.OpCode.Name, mono, XBrushes.Firebrick, new XPoint(x, y));
        gfx.DrawString(operands.Length > 44 ? string.Concat(operands.AsSpan(0, 41), "...") : operands,
            mono, XBrushes.DimGray, new XPoint(x + 26, y));

        y += 11;
        if (y > 740)
        {
            y = 140;
            x += 250;
        }
    }
}

// ----- page 3: the tally, and what the text does not say -----

var tally = document.AddPage();
using (var gfx = XGraphics.FromPdfPage(tally))
{
    var prose = new XTextFormatter(gfx);

    gfx.DrawString("By operator", heading, XBrushes.Black, new XPoint(50, 60));

    prose.DrawString(
        "The same content counted rather than listed, which is usually the more useful "
        + "view: it says at a glance whether a page is mostly text, mostly paths or mostly "
        + "graphics state, and a page that is unexpectedly large usually says so here.",
        body, XBrushes.Black, new XRect(50, 80, 495, 45));

    var counted = operators
        .GroupBy(op => op.OpCode.Name)
        .Select(group => new { Name = group.Key, Count = group.Count() })
        .OrderByDescending(entry => entry.Count)
        .ThenBy(entry => entry.Name)
        .ToList();

    (string Code, string Means)[] glossary =
    {
        ("q", "save the graphics state"), ("Q", "restore it"),
        ("cm", "concatenate a matrix"), ("re", "add a rectangle to the path"),
        ("m", "move to"), ("l", "line to"), ("c", "curve to"), ("h", "close the figure"),
        ("S", "stroke"), ("f", "fill"), ("f*", "fill, even-odd"), ("B", "fill and stroke"),
        ("n", "end the path, painting nothing"), ("W", "use the path as a clip"),
        ("W*", "clip, even-odd"), ("BT", "begin text"), ("ET", "end text"),
        ("Tf", "set font and size"), ("Td", "move the text position"),
        ("Tj", "show text"), ("TJ", "show text, with the parts moved apart"),
        ("rg", "set a non-stroking colour"), ("RG", "set a stroking colour"),
        ("w", "set the line width"), ("gs", "apply an extended graphics state"),
        ("J", "set the line cap"), ("j", "set the line join"), ("d", "set the dash"),
        ("Do", "paint an XObject"), ("M", "set the miter limit")
    };

    double y = 140;
    foreach (var entry in counted)
    {
        var means = glossary.FirstOrDefault(item => item.Code == entry.Name).Means;

        gfx.DrawString($"{entry.Count,4}", mono, XBrushes.Black, new XPoint(50, y));
        gfx.DrawString(entry.Name, mono, XBrushes.Firebrick, new XPoint(90, y));
        gfx.DrawString(means ?? "", body, XBrushes.DimGray, new XPoint(130, y));
        y += 13;
    }

    gfx.DrawString("Why the text is not readable", label, XBrushes.Black,
        new XPoint(50, y + 20));

    prose.DrawString(
        "The operands of a Tj are shown above as a byte count rather than as words, and "
        + "that is not the reader being coy. Fonts here are embedded as Identity-H by "
        + "default, so a show-text operator carries two-byte glyph identifiers rather than "
        + "characters - the face's own numbering, which means nothing without the font's "
        + "tables. Setting XPdfFontOptions.WinAnsiDefault writes readable string literals "
        + "instead, at the cost of the characters WinAnsi cannot represent. The Unicode "
        + "demo is where that trade is laid out.",
        body, XBrushes.Black, new XRect(50, y + 33, 495, 80));

    prose.DrawString(
        "Reading content back is how the test suite checks what the renderer wrote without "
        + "rasterizing anything - four of its helpers do exactly this. It is also the "
        + "quickest answer to \"why is my page blank\": a page whose operators are there "
        + "but whose coordinates are wrong looks identical to one that drew nothing, and "
        + "only one of the two says so here.",
        body, XBrushes.Black, new XRect(50, y + 118, 495, 60));
}