Add support for table areas and regions

.github/workflows/tests.yml

@@ -1,23 +0,0 @@
-name: tests
-
-on: [pull_request]
-
-jobs:
-  test:
-    runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        python-version: [3.6, 3.7, 3.8, 3.9]
-
-    steps:
-      - uses: actions/checkout@v2
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
-        with:
-          python-version: ${{ matrix.python-version }}
-      - name: Install camelot with dependencies
-        run: |
-          make install
-      - name: Test with pytest
-        run: |
-          make test

.travis.yml

@@ -0,0 +1,29 @@
+sudo: true
+language: python
+cache: pip
+addons:
+  apt:
+    update: true
+install:
+  - make install
+jobs:
+  include:
+    - stage: test
+      script:
+        - make test
+      python: '3.6'
+    - stage: test
+      script:
+        - make test
+      python: '3.7'
+      dist: xenial
+    - stage: test
+      script:
+        - make test
+      python: '3.8'
+      dist: xenial
+    - stage: coverage
+      python: '3.8'
+      script:
+        - make test
+        - codecov --verbose

HISTORY.md

@@ -4,31 +4,6 @@ Release History
 master
 ------
 
-0.9.0 (2021-06-15)
-------------------
-
-**Bugfixes**
-
-- Fix use of resolution argument to generate image with ghostscript. [#231](https://github.com/camelot-dev/camelot/pull/231) by [Tiago Samaha Cordeiro](https://github.com/tiagosamaha).
-- [#15](https://github.com/camelot-dev/camelot/issues/15) Fix duplicate strings being assigned to the same cell. [#206](https://github.com/camelot-dev/camelot/pull/206) by [Eduardo Gonzalez Lopez de Murillas](https://github.com/edugonza).
-- Save plot when filename is specified. [#121](https://github.com/camelot-dev/camelot/pull/121) by [Jens Diemer](https://github.com/jedie).
-- Close file streams explicitly. [#202](https://github.com/camelot-dev/camelot/pull/202) by [Martin Abente Lahaye](https://github.com/tchx84).
-- Use correct re.sub signature. [#186](https://github.com/camelot-dev/camelot/pull/186) by [pevisscher](https://github.com/pevisscher).
-- [#183](https://github.com/camelot-dev/camelot/issues/183) Fix UnicodeEncodeError when using Stream flavor by adding encoding kwarg to `to_html`. [#188](https://github.com/camelot-dev/camelot/pull/188) by [Stefano Fiorucci](https://github.com/anakin87).
-- [#179](https://github.com/camelot-dev/camelot/issues/179) Fix `max() arg is an empty sequence` error on PDFs with blank pages. [#189](https://github.com/camelot-dev/camelot/pull/189) by Vinayak Mehta.
-
-**Improvements**
-
-- Add `line_overlap` and `boxes_flow` to `LAParams`. [#219](https://github.com/camelot-dev/camelot/pull/219) by [Arnie97](https://github.com/Arnie97).
-- [Add bug report template.](https://github.com/camelot-dev/camelot/commit/0a3944e54d133b701edfe9c7546ff11289301ba8)
-- Move from [Travis to GitHub Actions](https://github.com/camelot-dev/camelot/pull/241).
-- Update `.readthedocs.yml` and [remove requirements.txt](https://github.com/camelot-dev/camelot/commit/7ab5db39d07baa4063f975e9e00f6073340e04c1#diff-cde814ef2f549dc093f5b8fc533b7e8f47e7b32a8081e0760e57d5c25a1139d9)
-
-**Documentation**
-
-- [#193](https://github.com/camelot-dev/camelot/issues/193) Add better checks to confirm proper installation of ghostscript. [#196](https://github.com/camelot-dev/camelot/pull/196) by [jimhall](https://github.com/jimhall).
-- Update `advanced.rst` plotting examples. [#119](https://github.com/camelot-dev/camelot/pull/119) by [Jens Diemer](https://github.com/jedie).
-
 0.8.2 (2020-07-27)
 ------------------
 

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2019-2021 Camelot Developers
+Copyright (c) 2019-2020 Camelot Developers
 Copyright (c) 2018-2019 Peeply Private Ltd (Singapore)
 
 Permission is hereby granted, free of charge, to any person obtaining a copy

README.md

@@ -4,10 +4,11 @@
 
 # Camelot: PDF Table Extraction for Humans
 
-![Build Status](https://github.com/camelot-dev/camelot/actions/workflows/tests.yml/badge.svg) [![Documentation Status](https://readthedocs.org/projects/camelot-py/badge/?version=master)](https://camelot-py.readthedocs.io/en/master/)
+[![Build Status](https://travis-ci.org/camelot-dev/camelot.svg?branch=master)](https://travis-ci.org/camelot-dev/camelot) [![Documentation Status](https://readthedocs.org/projects/camelot-py/badge/?version=master)](https://camelot-py.readthedocs.io/en/master/)
  [![codecov.io](https://codecov.io/github/camelot-dev/camelot/badge.svg?branch=master&service=github)](https://codecov.io/github/camelot-dev/camelot?branch=master)
  [![image](https://img.shields.io/pypi/v/camelot-py.svg)](https://pypi.org/project/camelot-py/) [![image](https://img.shields.io/pypi/l/camelot-py.svg)](https://pypi.org/project/camelot-py/) [![image](https://img.shields.io/pypi/pyversions/camelot-py.svg)](https://pypi.org/project/camelot-py/) [![Gitter chat](https://badges.gitter.im/camelot-dev/Lobby.png)](https://gitter.im/camelot-dev/Lobby)
-[![image](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)
+[![image](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black) [![image](https://img.shields.io/badge/continous%20quality-deepsource-lightgrey)](https://deepsource.io/gh/camelot-dev/camelot/?ref=repository-badge)
+
 
 **Camelot** is a Python library that can help you extract tables from PDFs!
 

camelot/__version__.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 
-VERSION = (0, 9, 0)
+VERSION = (0, 8, 2)
 PRERELEASE = None  # alpha, beta or rc
 REVISION = None
 

camelot/core.py

@@ -55,9 +55,7 @@ class TextEdge(object):
         x = round(self.x, 2)
         y0 = round(self.y0, 2)
         y1 = round(self.y1, 2)
-        return (
-            f"<TextEdge x={x} y0={y0} y1={y1} align={self.align} valid={self.is_valid}>"
-        )
+        return f"<TextEdge x={x} y0={y0} y1={y1} align={self.align} valid={self.is_valid}>"
 
     def update_coords(self, x, y0, edge_tol=50):
         """Updates the text edge's x and bottom y coordinates and sets
@@ -104,7 +102,8 @@ class TextEdges(object):
         return None
 
     def add(self, textline, align):
-        """Adds a new text edge to the current dict."""
+        """Adds a new text edge to the current dict.
+        """
         x = self.get_x_coord(textline, align)
         y0 = textline.y0
         y1 = textline.y1
@@ -112,7 +111,8 @@ class TextEdges(object):
         self._textedges[align].append(te)
 
     def update(self, textline):
-        """Updates an existing text edge in the current dict."""
+        """Updates an existing text edge in the current dict.
+        """
         for align in ["left", "right", "middle"]:
             x_coord = self.get_x_coord(textline, align)
             idx = self.find(x_coord, align)
@@ -304,7 +304,8 @@ class Cell(object):
 
     @property
     def bound(self):
-        """The number of sides on which the cell is bounded."""
+        """The number of sides on which the cell is bounded.
+        """
         return self.top + self.bottom + self.left + self.right
 
 
@@ -360,7 +361,8 @@ class Table(object):
 
     @property
     def data(self):
-        """Returns two-dimensional list of strings in table."""
+        """Returns two-dimensional list of strings in table.
+        """
         d = []
         for row in self.cells:
             d.append([cell.text.strip() for cell in row])
@@ -381,7 +383,8 @@ class Table(object):
         return report
 
     def set_all_edges(self):
-        """Sets all table edges to True."""
+        """Sets all table edges to True.
+        """
         for row in self.cells:
             for cell in row:
                 cell.left = cell.right = cell.top = cell.bottom = True
@@ -523,7 +526,8 @@ class Table(object):
         return self
 
     def set_border(self):
-        """Sets table border edges to True."""
+        """Sets table border edges to True.
+        """
         for r in range(len(self.rows)):
             self.cells[r][0].left = True
             self.cells[r][len(self.cols) - 1].right = True

camelot/ext/ghostscript/__init__.py

@@ -81,7 +81,8 @@ class __Ghostscript(object):
 
 
 def Ghostscript(*args, **kwargs):
-    """Factory function for setting up a Ghostscript instance"""
+    """Factory function for setting up a Ghostscript instance
+    """
     global __instance__
     # Ghostscript only supports a single instance
     if __instance__ is None:

camelot/handlers.py

@@ -6,7 +6,7 @@ import sys
 from PyPDF2 import PdfFileReader, PdfFileWriter
 
 from .core import TableList
-from .parsers import Stream, Lattice
+from .parsers import Lattice, Stream, LatticeOCR, StreamOCR
 from .utils import (
     TemporaryDirectory,
     get_page_layout,
@@ -163,12 +163,19 @@ class PDFHandler(object):
             List of tables found in PDF.
 
         """
+        parsers = {
+            "lattice": Lattice,
+            "stream": Stream,
+            "lattice_ocr": LatticeOCR,
+            "stream_ocr": StreamOCR,
+        }
+
         tables = []
         with TemporaryDirectory() as tempdir:
             for p in self.pages:
                 self._save_page(self.filepath, p, tempdir)
             pages = [os.path.join(tempdir, f"page-{p}.pdf") for p in self.pages]
-            parser = Lattice(**kwargs) if flavor == "lattice" else Stream(**kwargs)
+            parser = parsers[flavor](**kwargs)
             for p in pages:
                 t = parser.extract_tables(
                     p, suppress_stdout=suppress_stdout, layout_kwargs=layout_kwargs

camelot/io.py

@@ -98,9 +98,10 @@ def read_pdf(
     tables : camelot.core.TableList
 
     """
-    if flavor not in ["lattice", "stream"]:
+    if flavor not in ["lattice", "stream", "lattice_ocr", "stream_ocr"]:
         raise NotImplementedError(
-            "Unknown flavor specified." " Use either 'lattice' or 'stream'"
+            "Unknown flavor specified. Use one of the following: "
+            "'lattice', 'stream', 'lattice_ocr', 'stream_ocr'"
         )
 
     with warnings.catch_warnings():

camelot/parsers/__init__.py

@@ -1,4 +1,6 @@
 # -*- coding: utf-8 -*-
 
-from .stream import Stream
 from .lattice import Lattice
+from .stream import Stream
+from .lattice_ocr import LatticeOCR
+from .stream_ocr import StreamOCR

camelot/parsers/base.py

@@ -6,7 +6,8 @@ from ..utils import get_page_layout, get_text_objects
 
 
 class BaseParser(object):
-    """Defines a base parser."""
+    """Defines a base parser.
+    """
 
     def _generate_layout(self, filename, layout_kwargs):
         self.filename = filename

camelot/parsers/lattice.py

@@ -211,8 +211,8 @@ class Lattice(BaseParser):
         from ..ext.ghostscript import Ghostscript
 
         self.imagename = "".join([self.rootname, ".png"])
-        gs_call = "-q -sDEVICE=png16m -o {} -r{} {}".format(
-            self.imagename, self.resolution, self.filename
+        gs_call = "-q -sDEVICE=png16m -o {} -r300 {}".format(
+            self.imagename, self.filename
         )
         gs_call = gs_call.encode().split()
         null = open(os.devnull, "wb")

camelot/parsers/lattice_ocr.py

@@ -0,0 +1,243 @@
+# -*- coding: utf-8 -*-
+
+import os
+import copy
+import logging
+import subprocess
+
+try:
+    import easyocr
+except ImportError:
+    _HAS_EASYOCR = False
+else:
+    _HAS_EASYOCR = True
+
+import pandas as pd
+from PIL import Image
+
+from .base import BaseParser
+from ..core import Table
+from ..utils import TemporaryDirectory, merge_close_lines, scale_pdf, segments_in_bbox
+from ..image_processing import (
+    adaptive_threshold,
+    find_lines,
+    find_contours,
+    find_joints,
+)
+
+
+logger = logging.getLogger("camelot")
+
+
+class LatticeOCR(BaseParser):
+    def __init__(
+        self,
+        table_regions=None,
+        table_areas=None,
+        line_scale=15,
+        line_tol=2,
+        joint_tol=2,
+        threshold_blocksize=15,
+        threshold_constant=-2,
+        iterations=0,
+        resolution=300,
+    ):
+        self.table_regions = table_regions
+        self.table_areas = table_areas
+        self.line_scale = line_scale
+        self.line_tol = line_tol
+        self.joint_tol = joint_tol
+        self.threshold_blocksize = threshold_blocksize
+        self.threshold_constant = threshold_constant
+        self.iterations = iterations
+        self.resolution = resolution
+
+        if _HAS_EASYOCR:
+            self.reader = easyocr.Reader(['en'], gpu=False)
+        else:
+            raise ImportError("easyocr is required to run OCR on image-based PDFs.")
+
+    def _generate_image(self):
+        from ..ext.ghostscript import Ghostscript
+
+        self.imagename = "".join([self.rootname, ".png"])
+        gs_call = "-q -sDEVICE=png16m -o {} -r900 {}".format(
+            self.imagename, self.filename
+        )
+        gs_call = gs_call.encode().split()
+        null = open(os.devnull, "wb")
+        with Ghostscript(*gs_call, stdout=null) as gs:
+            pass
+        null.close()
+
+    def _generate_table_bbox(self):
+        def scale_areas(areas, scalers):
+            scaled_areas = []
+            for area in areas:
+                x1, y1, x2, y2 = area.split(",")
+                x1 = float(x1)
+                y1 = float(y1)
+                x2 = float(x2)
+                y2 = float(y2)
+                x1, y1, x2, y2 = scale_pdf((x1, y1, x2, y2), scalers)
+                scaled_areas.append((x1, y1, abs(x2 - x1), abs(y2 - y1)))
+            return scaled_areas
+
+        self.image, self.threshold = adaptive_threshold(
+            self.imagename, blocksize=self.threshold_blocksize, c=self.threshold_constant
+        )
+
+        image_width = self.image.shape[1]
+        image_height = self.image.shape[0]
+        image_width_scaler = image_width / float(self.pdf_width)
+        image_height_scaler = image_height / float(self.pdf_height)
+        image_scalers = (image_width_scaler, image_height_scaler, self.pdf_height)
+
+        if self.table_areas is None:
+            regions = None
+            if self.table_regions is not None:
+                regions = scale_areas(self.table_regions, image_scalers)
+
+            vertical_mask, vertical_segments = find_lines(
+                self.threshold,
+                regions=regions,
+                direction="vertical",
+                line_scale=self.line_scale,
+                iterations=self.iterations,
+            )
+            horizontal_mask, horizontal_segments = find_lines(
+                self.threshold,
+                regions=regions,
+                direction="horizontal",
+                line_scale=self.line_scale,
+                iterations=self.iterations,
+            )
+
+            contours = find_contours(vertical_mask, horizontal_mask)
+            table_bbox = find_joints(contours, vertical_mask, horizontal_mask)
+        else:
+            vertical_mask, vertical_segments = find_lines(
+                self.threshold,
+                direction="vertical",
+                line_scale=self.line_scale,
+                iterations=self.iterations,
+            )
+            horizontal_mask, horizontal_segments = find_lines(
+                self.threshold,
+                direction="horizontal",
+                line_scale=self.line_scale,
+                iterations=self.iterations,
+            )
+
+            areas = scale_areas(self.table_areas, image_scalers)
+            table_bbox = find_joints(areas, vertical_mask, horizontal_mask)
+
+        self.table_bbox_unscaled = copy.deepcopy(table_bbox)
+
+        self.table_bbox = table_bbox
+        self.vertical_segments = vertical_segments
+        self.horizontal_segments = horizontal_segments
+
+    def _generate_columns_and_rows(self, table_idx, tk):
+        cols, rows = zip(*self.table_bbox[tk])
+        cols, rows = list(cols), list(rows)
+        cols.extend([tk[0], tk[2]])
+        rows.extend([tk[1], tk[3]])
+        # sort horizontal and vertical segments
+        cols = merge_close_lines(sorted(cols), line_tol=self.line_tol)
+        rows = merge_close_lines(sorted(rows), line_tol=self.line_tol)
+        # make grid using x and y coord of shortlisted rows and cols
+        cols = [(cols[i], cols[i + 1]) for i in range(0, len(cols) - 1)]
+        rows = [(rows[i], rows[i + 1]) for i in range(0, len(rows) - 1)]
+
+        return cols, rows
+
+
+    def _generate_table(self, table_idx, cols, rows, **kwargs):
+        table = Table(cols, rows)
+        # set table edges to True using ver+hor lines
+        table = table.set_edges(self.vertical_segments, self.horizontal_segments, joint_tol=self.joint_tol)
+        # set table border edges to True
+        table = table.set_border()
+        # set spanning cells to True
+        table = table.set_span()
+
+        _seen = set()
+        for r_idx in range(len(table.cells)):
+            for c_idx in range(len(table.cells[r_idx])):
+                if (r_idx, c_idx) in _seen:
+                    continue
+
+                _seen.add((r_idx, c_idx))
+
+                _r_idx = r_idx
+                _c_idx = c_idx
+
+                if table.cells[r_idx][_c_idx].hspan:
+                    while not table.cells[r_idx][_c_idx].right:
+                        _c_idx += 1
+                        _seen.add((r_idx, _c_idx))
+
+                if table.cells[_r_idx][c_idx].vspan:
+                    while not table.cells[_r_idx][c_idx].bottom:
+                        _r_idx += 1
+                        _seen.add((_r_idx, c_idx))
+
+                for i in range(r_idx, _r_idx + 1):
+                    for j in range(c_idx, _c_idx + 1):
+                        _seen.add((i, j))
+
+                x1 = int(table.cells[r_idx][c_idx].x1)
+                y1 = int(table.cells[_r_idx][_c_idx].y1)
+
+                x2 = int(table.cells[_r_idx][_c_idx].x2)
+                y2 = int(table.cells[r_idx][c_idx].y2)
+
+                with TemporaryDirectory() as tempdir:
+                    temp_image_path = os.path.join(tempdir, f"{table_idx}_{r_idx}_{c_idx}.png")
+
+                    cell_image = Image.fromarray(self.image[y2:y1, x1:x2])
+                    cell_image.save(temp_image_path)
+
+                    text = self.reader.readtext(temp_image_path, detail=0)
+                    text = " ".join(text)
+
+                table.cells[r_idx][c_idx].text = text
+
+        data = table.data
+        table.df = pd.DataFrame(data)
+        table.shape = table.df.shape
+
+        table.flavor = "lattice_ocr"
+        table.accuracy = 0
+        table.whitespace = 0
+        table.order = table_idx + 1
+        table.page = int(os.path.basename(self.rootname).replace("page-", ""))
+
+        # for plotting
+        table._text = None
+        table._image = (self.image, self.table_bbox_unscaled)
+        table._segments = (self.vertical_segments, self.horizontal_segments)
+        table._textedges = None
+
+        return table
+
+    def extract_tables(self, filename, suppress_stdout=False, layout_kwargs={}):
+        self._generate_layout(filename, layout_kwargs)
+        if not suppress_stdout:
+            logger.info("Processing {}".format(os.path.basename(self.rootname)))
+
+        self._generate_image()
+        self._generate_table_bbox()
+
+        _tables = []
+        # sort tables based on y-coord
+        for table_idx, tk in enumerate(
+            sorted(self.table_bbox.keys(), key=lambda x: x[1], reverse=True)
+        ):
+            cols, rows = self._generate_columns_and_rows(table_idx, tk)
+            table = self._generate_table(table_idx, cols, rows)
+            table._bbox = tk
+            _tables.append(table)
+
+        return _tables

camelot/parsers/stream.py

@@ -65,7 +65,7 @@ class Stream(BaseParser):
         edge_tol=50,
         row_tol=2,
         column_tol=0,
-        **kwargs,
+        **kwargs
     ):
         self.table_regions = table_regions
         self.table_areas = table_areas
@@ -362,10 +362,10 @@ class Stream(BaseParser):
                     if len(elements):
                         ncols = max(set(elements), key=elements.count)
                     else:
-                        warnings.warn(f"No tables found in table area {table_idx + 1}")
-                cols = [
-                    (t.x0, t.x1) for r in rows_grouped if len(r) == ncols for t in r
-                ]
+                        warnings.warn(
+                            f"No tables found in table area {table_idx + 1}"
+                        )
+                cols = [(t.x0, t.x1) for r in rows_grouped if len(r) == ncols for t in r]
                 cols = self._merge_columns(sorted(cols), column_tol=self.column_tol)
                 inner_text = []
                 for i in range(1, len(cols)):

camelot/parsers/stream_ocr.py

@@ -0,0 +1,7 @@
+# -*- coding: utf-8 -*-
+
+from .base import BaseParser
+
+
+class StreamOCR(BaseParser):
+    pass

camelot/plotting.py

@@ -34,9 +34,13 @@ class PlotMethods(object):
             raise ImportError("matplotlib is required for plotting.")
 
         if table.flavor == "lattice" and kind in ["textedge"]:
-            raise NotImplementedError(f"Lattice flavor does not support kind='{kind}'")
+            raise NotImplementedError(
+                f"Lattice flavor does not support kind='{kind}'"
+            )
         elif table.flavor == "stream" and kind in ["joint", "line"]:
-            raise NotImplementedError(f"Stream flavor does not support kind='{kind}'")
+            raise NotImplementedError(
+                f"Stream flavor does not support kind='{kind}'"
+            )
 
         plot_method = getattr(self, kind)
         fig = plot_method(table)

camelot/utils.py

@@ -93,7 +93,6 @@ def download_url(url):
     return filepath
 
 
-stream_kwargs = ["columns", "edge_tol", "row_tol", "column_tol"]
 lattice_kwargs = [
     "process_background",
     "line_scale",
@@ -106,6 +105,7 @@ lattice_kwargs = [
     "iterations",
     "resolution",
 ]
+stream_kwargs = ["columns", "edge_tol", "row_tol", "column_tol"]
 
 
 def validate_input(kwargs, flavor="lattice"):
@@ -116,14 +116,14 @@ def validate_input(kwargs, flavor="lattice"):
                 f"{','.join(sorted(isec))} cannot be used with flavor='{flavor}'"
             )
 
-    if flavor == "lattice":
+    if flavor in ["lattice", "lattice_ocr"]:
         check_intersection(stream_kwargs, kwargs)
     else:
         check_intersection(lattice_kwargs, kwargs)
 
 
 def remove_extra(kwargs, flavor="lattice"):
-    if flavor == "lattice":
+    if flavor in ["lattice", "lattice_ocr"]:
         for key in kwargs.keys():
             if key in stream_kwargs:
                 kwargs.pop(key)
@@ -838,27 +838,23 @@ def compute_whitespace(d):
 
 def get_page_layout(
     filename,
-    line_overlap=0.5,
     char_margin=1.0,
     line_margin=0.5,
     word_margin=0.1,
-    boxes_flow=0.5,
     detect_vertical=True,
     all_texts=True,
 ):
     """Returns a PDFMiner LTPage object and page dimension of a single
-    page pdf. To get the definitions of kwargs, see
-    https://pdfminersix.rtfd.io/en/latest/reference/composable.html.
+    page pdf. See https://euske.github.io/pdfminer/ to get definitions
+    of kwargs.
 
     Parameters
     ----------
     filename : string
         Path to pdf file.
-    line_overlap : float
     char_margin : float
     line_margin : float
     word_margin : float
-    boxes_flow : float
     detect_vertical : bool
     all_texts : bool
 
@@ -874,15 +870,11 @@ def get_page_layout(
         parser = PDFParser(f)
         document = PDFDocument(parser)
         if not document.is_extractable:
-            raise PDFTextExtractionNotAllowed(
-                f"Text extraction is not allowed: {filename}"
-            )
+            raise PDFTextExtractionNotAllowed(f"Text extraction is not allowed: {filename}")
         laparams = LAParams(
-            line_overlap=line_overlap,
             char_margin=char_margin,
             line_margin=line_margin,
             word_margin=word_margin,
-            boxes_flow=boxes_flow,
             detect_vertical=detect_vertical,
             all_texts=all_texts,
         )

docs/_themes/flask_theme_support.py

@@ -1,19 +1,7 @@
 # flasky pygments style based on tango style
 from pygments.style import Style
-from pygments.token import (
-    Keyword,
-    Name,
-    Comment,
-    String,
-    Error,
-    Number,
-    Operator,
-    Generic,
-    Whitespace,
-    Punctuation,
-    Other,
-    Literal,
-)
+from pygments.token import Keyword, Name, Comment, String, Error, \
+     Number, Operator, Generic, Whitespace, Punctuation, Other, Literal
 
 
 class FlaskyStyle(Style):
@@ -23,67 +11,76 @@ class FlaskyStyle(Style):
     styles = {
         # No corresponding class for the following:
         # Text:                    "", # class:  ''
-        Whitespace: "underline #f8f8f8",  # class: 'w'
-        Error: "#a40000 border:#ef2929",  # class: 'err'
-        Other: "#000000",  # class 'x'
-        Comment: "italic #8f5902",  # class: 'c'
-        Comment.Preproc: "noitalic",  # class: 'cp'
-        Keyword: "bold #004461",  # class: 'k'
-        Keyword.Constant: "bold #004461",  # class: 'kc'
-        Keyword.Declaration: "bold #004461",  # class: 'kd'
-        Keyword.Namespace: "bold #004461",  # class: 'kn'
-        Keyword.Pseudo: "bold #004461",  # class: 'kp'
-        Keyword.Reserved: "bold #004461",  # class: 'kr'
-        Keyword.Type: "bold #004461",  # class: 'kt'
-        Operator: "#582800",  # class: 'o'
-        Operator.Word: "bold #004461",  # class: 'ow' - like keywords
-        Punctuation: "bold #000000",  # class: 'p'
+        Whitespace:                "underline #f8f8f8",       # class: 'w'
+        Error:                     "#a40000 border:#ef2929",  # class: 'err'
+        Other:                     "#000000",                 # class 'x'
+
+        Comment:                   "italic #8f5902",  # class: 'c'
+        Comment.Preproc:           "noitalic",        # class: 'cp'
+
+        Keyword:                   "bold #004461",    # class: 'k'
+        Keyword.Constant:          "bold #004461",    # class: 'kc'
+        Keyword.Declaration:       "bold #004461",    # class: 'kd'
+        Keyword.Namespace:         "bold #004461",    # class: 'kn'
+        Keyword.Pseudo:            "bold #004461",    # class: 'kp'
+        Keyword.Reserved:          "bold #004461",    # class: 'kr'
+        Keyword.Type:              "bold #004461",    # class: 'kt'
+
+        Operator:                  "#582800",   # class: 'o'
+        Operator.Word:             "bold #004461",   # class: 'ow' - like keywords
+
+        Punctuation:               "bold #000000",   # class: 'p'
+
         # because special names such as Name.Class, Name.Function, etc.
         # are not recognized as such later in the parsing, we choose them
         # to look the same as ordinary variables.
-        Name: "#000000",  # class: 'n'
-        Name.Attribute: "#c4a000",  # class: 'na' - to be revised
-        Name.Builtin: "#004461",  # class: 'nb'
-        Name.Builtin.Pseudo: "#3465a4",  # class: 'bp'
-        Name.Class: "#000000",  # class: 'nc' - to be revised
-        Name.Constant: "#000000",  # class: 'no' - to be revised
-        Name.Decorator: "#888",  # class: 'nd' - to be revised
-        Name.Entity: "#ce5c00",  # class: 'ni'
-        Name.Exception: "bold #cc0000",  # class: 'ne'
-        Name.Function: "#000000",  # class: 'nf'
-        Name.Property: "#000000",  # class: 'py'
-        Name.Label: "#f57900",  # class: 'nl'
-        Name.Namespace: "#000000",  # class: 'nn' - to be revised
-        Name.Other: "#000000",  # class: 'nx'
-        Name.Tag: "bold #004461",  # class: 'nt' - like a keyword
-        Name.Variable: "#000000",  # class: 'nv' - to be revised
-        Name.Variable.Class: "#000000",  # class: 'vc' - to be revised
-        Name.Variable.Global: "#000000",  # class: 'vg' - to be revised
-        Name.Variable.Instance: "#000000",  # class: 'vi' - to be revised
-        Number: "#990000",  # class: 'm'
-        Literal: "#000000",  # class: 'l'
-        Literal.Date: "#000000",  # class: 'ld'
-        String: "#4e9a06",  # class: 's'
-        String.Backtick: "#4e9a06",  # class: 'sb'
-        String.Char: "#4e9a06",  # class: 'sc'
-        String.Doc: "italic #8f5902",  # class: 'sd' - like a comment
-        String.Double: "#4e9a06",  # class: 's2'
-        String.Escape: "#4e9a06",  # class: 'se'
-        String.Heredoc: "#4e9a06",  # class: 'sh'
-        String.Interpol: "#4e9a06",  # class: 'si'
-        String.Other: "#4e9a06",  # class: 'sx'
-        String.Regex: "#4e9a06",  # class: 'sr'
-        String.Single: "#4e9a06",  # class: 's1'
-        String.Symbol: "#4e9a06",  # class: 'ss'
-        Generic: "#000000",  # class: 'g'
-        Generic.Deleted: "#a40000",  # class: 'gd'
-        Generic.Emph: "italic #000000",  # class: 'ge'
-        Generic.Error: "#ef2929",  # class: 'gr'
-        Generic.Heading: "bold #000080",  # class: 'gh'
-        Generic.Inserted: "#00A000",  # class: 'gi'
-        Generic.Output: "#888",  # class: 'go'
-        Generic.Prompt: "#745334",  # class: 'gp'
-        Generic.Strong: "bold #000000",  # class: 'gs'
-        Generic.Subheading: "bold #800080",  # class: 'gu'
-        Generic.Traceback: "bold #a40000",  # class: 'gt'
+        Name:                      "#000000",         # class: 'n'
+        Name.Attribute:            "#c4a000",         # class: 'na' - to be revised
+        Name.Builtin:              "#004461",         # class: 'nb'
+        Name.Builtin.Pseudo:       "#3465a4",         # class: 'bp'
+        Name.Class:                "#000000",         # class: 'nc' - to be revised
+        Name.Constant:             "#000000",         # class: 'no' - to be revised
+        Name.Decorator:            "#888",            # class: 'nd' - to be revised
+        Name.Entity:               "#ce5c00",         # class: 'ni'
+        Name.Exception:            "bold #cc0000",    # class: 'ne'
+        Name.Function:             "#000000",         # class: 'nf'
+        Name.Property:             "#000000",         # class: 'py'
+        Name.Label:                "#f57900",         # class: 'nl'
+        Name.Namespace:            "#000000",         # class: 'nn' - to be revised
+        Name.Other:                "#000000",         # class: 'nx'
+        Name.Tag:                  "bold #004461",    # class: 'nt' - like a keyword
+        Name.Variable:             "#000000",         # class: 'nv' - to be revised
+        Name.Variable.Class:       "#000000",         # class: 'vc' - to be revised
+        Name.Variable.Global:      "#000000",         # class: 'vg' - to be revised
+        Name.Variable.Instance:    "#000000",         # class: 'vi' - to be revised
+
+        Number:                    "#990000",         # class: 'm'
+
+        Literal:                   "#000000",         # class: 'l'
+        Literal.Date:              "#000000",         # class: 'ld'
+
+        String:                    "#4e9a06",         # class: 's'
+        String.Backtick:           "#4e9a06",         # class: 'sb'
+        String.Char:               "#4e9a06",         # class: 'sc'
+        String.Doc:                "italic #8f5902",  # class: 'sd' - like a comment
+        String.Double:             "#4e9a06",         # class: 's2'
+        String.Escape:             "#4e9a06",         # class: 'se'
+        String.Heredoc:            "#4e9a06",         # class: 'sh'
+        String.Interpol:           "#4e9a06",         # class: 'si'
+        String.Other:              "#4e9a06",         # class: 'sx'
+        String.Regex:              "#4e9a06",         # class: 'sr'
+        String.Single:             "#4e9a06",         # class: 's1'
+        String.Symbol:             "#4e9a06",         # class: 'ss'
+
+        Generic:                   "#000000",         # class: 'g'
+        Generic.Deleted:           "#a40000",         # class: 'gd'
+        Generic.Emph:              "italic #000000",  # class: 'ge'
+        Generic.Error:             "#ef2929",         # class: 'gr'
+        Generic.Heading:           "bold #000080",    # class: 'gh'
+        Generic.Inserted:          "#00A000",         # class: 'gi'
+        Generic.Output:            "#888",            # class: 'go'
+        Generic.Prompt:            "#745334",         # class: 'gp'
+        Generic.Strong:            "bold #000000",    # class: 'gs'
+        Generic.Subheading:        "bold #800080",    # class: 'gu'
+        Generic.Traceback:         "bold #a40000",    # class: 'gt'
     }

docs/conf.py

@@ -22,8 +22,8 @@ import sys
 # sys.path.insert(0, os.path.abspath('..'))
 
 # Insert Camelot's path into the system.
-sys.path.insert(0, os.path.abspath(".."))
-sys.path.insert(0, os.path.abspath("_themes"))
+sys.path.insert(0, os.path.abspath('..'))
+sys.path.insert(0, os.path.abspath('_themes'))
 
 import camelot
 
@@ -38,33 +38,33 @@ import camelot
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-    "sphinx.ext.autodoc",
-    "sphinx.ext.napoleon",
-    "sphinx.ext.intersphinx",
-    "sphinx.ext.todo",
-    "sphinx.ext.viewcode",
+    'sphinx.ext.autodoc',
+    'sphinx.ext.napoleon',
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.todo',
+    'sphinx.ext.viewcode',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ["_templates"]
+templates_path = ['_templates']
 
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
 # source_suffix = ['.rst', '.md']
-source_suffix = ".rst"
+source_suffix = '.rst'
 
 # The encoding of source files.
 #
 # source_encoding = 'utf-8-sig'
 
 # The master toctree document.
-master_doc = "index"
+master_doc = 'index'
 
 # General information about the project.
-project = u"Camelot"
-copyright = u"2021, Camelot Developers"
-author = u"Vinayak Mehta"
+project = u'Camelot'
+copyright = u'2020, Camelot Developers'
+author = u'Vinayak Mehta'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -94,7 +94,7 @@ language = None
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This patterns also effect to html_static_path and html_extra_path
-exclude_patterns = ["_build"]
+exclude_patterns = ['_build']
 
 # The reST default role (used for this markup: `text`) to use for all
 # documents.
@@ -114,7 +114,7 @@ add_module_names = True
 # show_authors = False
 
 # The name of the Pygments (syntax highlighting) style to use.
-pygments_style = "flask_theme_support.FlaskyStyle"
+pygments_style = 'flask_theme_support.FlaskyStyle'
 
 # A list of ignored prefixes for module index sorting.
 # modindex_common_prefix = []
@@ -130,18 +130,18 @@ todo_include_todos = True
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-html_theme = "alabaster"
+html_theme = 'alabaster'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
 html_theme_options = {
-    "show_powered_by": False,
-    "github_user": "camelot-dev",
-    "github_repo": "camelot",
-    "github_banner": True,
-    "show_related": False,
-    "note_bg": "#FFF59C",
+    'show_powered_by': False,
+    'github_user': 'camelot-dev',
+    'github_repo': 'camelot',
+    'github_banner': True,
+    'show_related': False,
+    'note_bg': '#FFF59C'
 }
 
 # Add any paths that contain custom themes here, relative to this directory.
@@ -164,12 +164,12 @@ html_theme_options = {
 # The name of an image file (relative to this directory) to use as a favicon of
 # the docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
 # pixels large.
-html_favicon = "_static/favicon.ico"
+html_favicon = '_static/favicon.ico'
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ["_static"]
+html_static_path = ['_static']
 
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied
@@ -189,21 +189,10 @@ html_use_smartypants = True
 
 # Custom sidebar templates, maps document names to template names.
 html_sidebars = {
-    "index": [
-        "sidebarintro.html",
-        "relations.html",
-        "sourcelink.html",
-        "searchbox.html",
-        "hacks.html",
-    ],
-    "**": [
-        "sidebarlogo.html",
-        "localtoc.html",
-        "relations.html",
-        "sourcelink.html",
-        "searchbox.html",
-        "hacks.html",
-    ],
+    'index': ['sidebarintro.html', 'relations.html', 'sourcelink.html',
+              'searchbox.html', 'hacks.html'],
+    '**': ['sidebarlogo.html', 'localtoc.html', 'relations.html',
+           'sourcelink.html', 'searchbox.html', 'hacks.html']
 }
 
 # Additional templates that should be rendered to pages, maps page names to
@@ -260,30 +249,34 @@ html_show_copyright = True
 # html_search_scorer = 'scorer.js'
 
 # Output file base name for HTML help builder.
-htmlhelp_basename = "Camelotdoc"
+htmlhelp_basename = 'Camelotdoc'
 
 # -- Options for LaTeX output ---------------------------------------------
 
 latex_elements = {
-    # The paper size ('letterpaper' or 'a4paper').
-    #
-    # 'papersize': 'letterpaper',
-    # The font size ('10pt', '11pt' or '12pt').
-    #
-    # 'pointsize': '10pt',
-    # Additional stuff for the LaTeX preamble.
-    #
-    # 'preamble': '',
-    # Latex figure (float) alignment
-    #
-    # 'figure_align': 'htbp',
+     # The paper size ('letterpaper' or 'a4paper').
+     #
+     # 'papersize': 'letterpaper',
+
+     # The font size ('10pt', '11pt' or '12pt').
+     #
+     # 'pointsize': '10pt',
+
+     # Additional stuff for the LaTeX preamble.
+     #
+     # 'preamble': '',
+
+     # Latex figure (float) alignment
+     #
+     # 'figure_align': 'htbp',
 }
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-    (master_doc, "Camelot.tex", u"Camelot Documentation", u"Vinayak Mehta", "manual"),
+    (master_doc, 'Camelot.tex', u'Camelot Documentation',
+     u'Vinayak Mehta', 'manual'),
 ]
 
 # The name of an image file (relative to this directory) to place at the top of
@@ -323,7 +316,10 @@ latex_documents = [
 
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
-man_pages = [(master_doc, "Camelot", u"Camelot Documentation", [author], 1)]
+man_pages = [
+    (master_doc, 'Camelot', u'Camelot Documentation',
+     [author], 1)
+]
 
 # If true, show URL addresses after external links.
 #
@@ -336,15 +332,9 @@ man_pages = [(master_doc, "Camelot", u"Camelot Documentation", [author], 1)]
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-    (
-        master_doc,
-        "Camelot",
-        u"Camelot Documentation",
-        author,
-        "Camelot",
-        "One line description of project.",
-        "Miscellaneous",
-    ),
+    (master_doc, 'Camelot', u'Camelot Documentation',
+     author, 'Camelot', 'One line description of project.',
+     'Miscellaneous'),
 ]
 
 # Documents to append as an appendix to all manuals.
@@ -366,6 +356,6 @@ texinfo_documents = [
 
 # Example configuration for intersphinx: refer to the Python standard library.
 intersphinx_mapping = {
-    "https://docs.python.org/2": None,
-    "http://pandas.pydata.org/pandas-docs/stable": None,
+    'https://docs.python.org/2': None,
+    'http://pandas.pydata.org/pandas-docs/stable': None
 }

docs/user/advanced.rst

@@ -618,7 +618,7 @@ Tweak layout generation
 
 Camelot is built on top of PDFMiner's functionality of grouping characters on a page into words and sentences. In some cases (such as `#170 <https://github.com/camelot-dev/camelot/issues/170>`_ and `#215 <https://github.com/camelot-dev/camelot/issues/215>`_), PDFMiner can group characters that should belong to the same sentence into separate sentences.
 
-To deal with such cases, you can tweak PDFMiner's `LAParams kwargs <https://github.com/euske/pdfminer/blob/master/pdfminer/layout.py#L33>`_ to improve layout generation, by passing the keyword arguments as a dict using ``layout_kwargs`` in :meth:`read_pdf() <camelot.read_pdf>`. To know more about the parameters you can tweak, you can check out `PDFMiner docs <https://pdfminersix.rtfd.io/en/latest/reference/composable.html>`_.
+To deal with such cases, you can tweak PDFMiner's `LAParams kwargs <https://github.com/euske/pdfminer/blob/master/pdfminer/layout.py#L33>`_ to improve layout generation, by passing the keyword arguments as a dict using ``layout_kwargs`` in :meth:`read_pdf() <camelot.read_pdf>`. To know more about the parameters you can tweak, you can check out `PDFMiner docs <https://euske.github.io/pdfminer/>`_.
 
 ::
 

setup.py

@@ -6,76 +6,80 @@ from setuptools import find_packages
 
 here = os.path.abspath(os.path.dirname(__file__))
 about = {}
-with open(os.path.join(here, "camelot", "__version__.py"), "r") as f:
+with open(os.path.join(here, 'camelot', '__version__.py'), 'r') as f:
     exec(f.read(), about)
 
-with open("README.md", "r") as f:
+with open('README.md', 'r') as f:
     readme = f.read()
 
 
 requires = [
-    "chardet>=3.0.4",
-    "click>=6.7",
-    "numpy>=1.13.3",
-    "openpyxl>=2.5.8",
-    "pandas>=0.23.4",
-    "pdfminer.six>=20200726",
-    "PyPDF2>=1.26.0",
+    'chardet>=3.0.4',
+    'click>=6.7',
+    'numpy>=1.13.3',
+    'openpyxl>=2.5.8',
+    'pandas>=0.23.4',
+    'pdfminer.six>=20200726',
+    'PyPDF2>=1.26.0'
 ]
 
-cv_requires = ["opencv-python>=3.4.2.17"]
+cv_requires = [
+    'opencv-python>=3.4.2.17'
+]
+
+ocr_requires = [
+    'easyocr>=1.1.10'
+]
 
 plot_requires = [
-    "matplotlib>=2.2.3",
+    'matplotlib>=2.2.3',
 ]
 
 dev_requires = [
-    "codecov>=2.0.15",
-    "pytest>=5.4.3",
-    "pytest-cov>=2.10.0",
-    "pytest-mpl>=0.11",
-    "pytest-runner>=5.2",
-    "Sphinx>=3.1.2",
+    'codecov>=2.0.15',
+    'pytest>=5.4.3',
+    'pytest-cov>=2.10.0',
+    'pytest-mpl>=0.11',
+    'pytest-runner>=5.2',
+    'Sphinx>=3.1.2'
 ]
 
-all_requires = cv_requires + plot_requires
+all_requires = cv_requires + ocr_requires + plot_requires
 dev_requires = dev_requires + all_requires
 
 
 def setup_package():
-    metadata = dict(
-        name=about["__title__"],
-        version=about["__version__"],
-        description=about["__description__"],
-        long_description=readme,
-        long_description_content_type="text/markdown",
-        url=about["__url__"],
-        author=about["__author__"],
-        author_email=about["__author_email__"],
-        license=about["__license__"],
-        packages=find_packages(exclude=("tests",)),
-        install_requires=requires,
-        extras_require={
-            "all": all_requires,
-            "cv": cv_requires,
-            "dev": dev_requires,
-            "plot": plot_requires,
-        },
-        entry_points={
-            "console_scripts": [
-                "camelot = camelot.cli:cli",
-            ],
-        },
-        classifiers=[
-            # Trove classifiers
-            # Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers
-            "License :: OSI Approved :: MIT License",
-            "Programming Language :: Python :: 3.6",
-            "Programming Language :: Python :: 3.7",
-            "Programming Language :: Python :: 3.8",
-            "Programming Language :: Python :: 3.9",
-        ],
-    )
+    metadata = dict(name=about['__title__'],
+                    version=about['__version__'],
+                    description=about['__description__'],
+                    long_description=readme,
+                    long_description_content_type="text/markdown",
+                    url=about['__url__'],
+                    author=about['__author__'],
+                    author_email=about['__author_email__'],
+                    license=about['__license__'],
+                    packages=find_packages(exclude=('tests',)),
+                    install_requires=requires,
+                    extras_require={
+                        'cv': cv_requires,
+                        'ocr': ocr_requires,
+                        'plot': plot_requires,
+                        'all': all_requires,
+                        'dev': dev_requires,
+                    },
+                    entry_points={
+                        'console_scripts': [
+                            'camelot = camelot.cli:cli',
+                        ],
+                    },
+                    classifiers=[
+                        # Trove classifiers
+                        # Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers
+                        'License :: OSI Approved :: MIT License',
+                        'Programming Language :: Python :: 3.6',
+                        'Programming Language :: Python :: 3.7',
+                        'Programming Language :: Python :: 3.8'
+                    ])
 
     try:
         from setuptools import setup
@@ -85,5 +89,5 @@ def setup_package():
     setup(**metadata)
 
 
-if __name__ == "__main__":
+if __name__ == '__main__':
     setup_package()

tests/data.py

@@ -2800,467 +2800,49 @@ data_stream_layout_kwargs = [
 ]
 
 data_stream_duplicated_text = [
-    [
-        "",
-        "2012 BETTER VARIETIES Harvest Report for Minnesota Central  [ MNCE ]",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "ALL SEASON TEST",
-    ],
-    [
-        "",
-        "Doug Toreen, Renville County, MN 55310          [ BIRD ISLAND ]",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "1.3 - 2.0 MAT. GROUP",
-    ],
-    [
-        "PREV. CROP/HERB:",
-        "Corn / Surpass, Roundup",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "S2MNCE01",
-    ],
-    [
-        "SOIL DESCRIPTION:",
-        "",
-        "Canisteo clay loam, mod. well drained, non-irrigated",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-    ],
-    [
-        "SOIL CONDITIONS:",
-        "",
-        "High P, high K, 6.7 pH, 3.9% OM, Low SCN",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        '30" ROW SPACING',
-    ],
-    [
-        "TILLAGE/CULTIVATION:",
-        "conventional w/ fall till",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-    ],
-    ["PEST MANAGEMENT:", "Roundup twice", "", "", "", "", "", "", "", "", ""],
-    [
-        "SEEDED - RATE:",
-        "May 15",
-        "140 000 /A",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "TOP 30 for YIELD of 63 TESTED",
-        "",
-    ],
-    [
-        "HARVESTED - STAND:",
-        "Oct 3",
-        "122 921 /A",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "AVERAGE of (3) REPLICATIONS",
-        "",
-    ],
-    ["", "", "", "", "SCN", "Seed", "Yield", "Moisture", "Lodging", "Stand", "Gross"],
-    [
-        "Company/Brand",
-        "Product/Brand†",
-        "Technol.†",
-        "Mat.",
-        "Resist.",
-        "Trmt.†",
-        "Bu/A",
-        "%",
-        "%",
-        "(x 1000)",
-        "Income",
-    ],
-    [
-        "Kruger",
-        "K2 1901",
-        "RR2Y",
-        "1.9",
-        "R",
-        "Ac,PV",
-        "56.4",
-        "7.6",
-        "0",
-        "126.3",
-        "$846",
-    ],
-    [
-        "Stine",
-        "19RA02 §",
-        "RR2Y",
-        "1.9",
-        "R",
-        "CMB",
-        "55.3",
-        "7.6",
-        "0",
-        "120.0",
-        "$830",
-    ],
-    [
-        "Wensman",
-        "W 3190NR2",
-        "RR2Y",
-        "1.9",
-        "R",
-        "Ac",
-        "54.5",
-        "7.6",
-        "0",
-        "119.5",
-        "$818",
-    ],
-    ["Hefty", "H17Y12", "RR2Y", "1.7", "MR", "I", "53.7", "7.7", "0", "124.4", "$806"],
-    [
-        "Dyna-Gro",
-        "S15RY53",
-        "RR2Y",
-        "1.5",
-        "R",
-        "Ac",
-        "53.6",
-        "7.7",
-        "0",
-        "126.8",
-        "$804",
-    ],
-    [
-        "LG Seeds",
-        "C2050R2",
-        "RR2Y",
-        "2.1",
-        "R",
-        "Ac",
-        "53.6",
-        "7.7",
-        "0",
-        "123.9",
-        "$804",
-    ],
-    [
-        "Titan Pro",
-        "19M42",
-        "RR2Y",
-        "1.9",
-        "R",
-        "CMB",
-        "53.6",
-        "7.7",
-        "0",
-        "121.0",
-        "$804",
-    ],
-    [
-        "Stine",
-        "19RA02 (2) §",
-        "RR2Y",
-        "1.9",
-        "R",
-        "CMB",
-        "53.4",
-        "7.7",
-        "0",
-        "123.9",
-        "$801",
-    ],
-    [
-        "Asgrow",
-        "AG1832 §",
-        "RR2Y",
-        "1.8",
-        "MR",
-        "Ac,PV",
-        "52.9",
-        "7.7",
-        "0",
-        "122.0",
-        "$794",
-    ],
-    [
-        "Prairie Brand",
-        "PB-1566R2",
-        "RR2Y",
-        "1.5",
-        "R",
-        "CMB",
-        "52.8",
-        "7.7",
-        "0",
-        "122.9",
-        "$792",
-    ],
-    [
-        "Channel",
-        "1901R2",
-        "RR2Y",
-        "1.9",
-        "R",
-        "Ac,PV",
-        "52.8",
-        "7.6",
-        "0",
-        "123.4",
-        "$791",
-    ],
-    [
-        "Titan Pro",
-        "20M1",
-        "RR2Y",
-        "2.0",
-        "R",
-        "Am",
-        "52.5",
-        "7.5",
-        "0",
-        "124.4",
-        "$788",
-    ],
-    [
-        "Kruger",
-        "K2-2002",
-        "RR2Y",
-        "2.0",
-        "R",
-        "Ac,PV",
-        "52.4",
-        "7.9",
-        "0",
-        "125.4",
-        "$786",
-    ],
-    [
-        "Channel",
-        "1700R2",
-        "RR2Y",
-        "1.7",
-        "R",
-        "Ac,PV",
-        "52.3",
-        "7.9",
-        "0",
-        "123.9",
-        "$784",
-    ],
-    ["Hefty", "H16Y11", "RR2Y", "1.6", "MR", "I", "51.4", "7.6", "0", "123.9", "$771"],
-    [
-        "Anderson",
-        "162R2Y",
-        "RR2Y",
-        "1.6",
-        "R",
-        "None",
-        "51.3",
-        "7.5",
-        "0",
-        "119.5",
-        "$770",
-    ],
-    [
-        "Titan Pro",
-        "15M22",
-        "RR2Y",
-        "1.5",
-        "R",
-        "CMB",
-        "51.3",
-        "7.8",
-        "0",
-        "125.4",
-        "$769",
-    ],
-    [
-        "Dairyland",
-        "DSR-1710R2Y",
-        "RR2Y",
-        "1.7",
-        "R",
-        "CMB",
-        "51.3",
-        "7.7",
-        "0",
-        "122.0",
-        "$769",
-    ],
-    ["Hefty", "H20R3", "RR2Y", "2.0", "MR", "I", "50.5", "8.2", "0", "121.0", "$757"],
-    [
-        "Prairie Brand",
-        "PB 1743R2",
-        "RR2Y",
-        "1.7",
-        "R",
-        "CMB",
-        "50.2",
-        "7.7",
-        "0",
-        "125.8",
-        "$752",
-    ],
-    [
-        "Gold Country",
-        "1741",
-        "RR2Y",
-        "1.7",
-        "R",
-        "Ac",
-        "50.1",
-        "7.8",
-        "0",
-        "123.9",
-        "$751",
-    ],
-    [
-        "Trelay",
-        "20RR43",
-        "RR2Y",
-        "2.0",
-        "R",
-        "Ac,Ex",
-        "49.9",
-        "7.6",
-        "0",
-        "127.8",
-        "$749",
-    ],
-    ["Hefty", "H14R3", "RR2Y", "1.4", "MR", "I", "49.7", "7.7", "0", "122.9", "$746"],
-    [
-        "Prairie Brand",
-        "PB-2099NRR2",
-        "RR2Y",
-        "2.0",
-        "R",
-        "CMB",
-        "49.6",
-        "7.8",
-        "0",
-        "126.3",
-        "$743",
-    ],
-    [
-        "Wensman",
-        "W 3174NR2",
-        "RR2Y",
-        "1.7",
-        "R",
-        "Ac",
-        "49.3",
-        "7.6",
-        "0",
-        "122.5",
-        "$740",
-    ],
-    [
-        "Kruger",
-        "K2 1602",
-        "RR2Y",
-        "1.6",
-        "R",
-        "Ac,PV",
-        "48.7",
-        "7.6",
-        "0",
-        "125.4",
-        "$731",
-    ],
-    [
-        "NK Brand",
-        "S18-C2 §",
-        "RR2Y",
-        "1.8",
-        "R",
-        "CMB",
-        "48.7",
-        "7.7",
-        "0",
-        "126.8",
-        "$731",
-    ],
-    [
-        "Kruger",
-        "K2 1902",
-        "RR2Y",
-        "1.9",
-        "R",
-        "Ac,PV",
-        "48.7",
-        "7.5",
-        "0",
-        "124.4",
-        "$730",
-    ],
-    [
-        "Prairie Brand",
-        "PB-1823R2",
-        "RR2Y",
-        "1.8",
-        "R",
-        "None",
-        "48.5",
-        "7.6",
-        "0",
-        "121.0",
-        "$727",
-    ],
-    [
-        "Gold Country",
-        "1541",
-        "RR2Y",
-        "1.5",
-        "R",
-        "Ac",
-        "48.4",
-        "7.6",
-        "0",
-        "110.4",
-        "$726",
-    ],
-    ["", "", "", "", "", "Test Average  =", "47.6", "7.7", "0", "122.9", "$713"],
-    ["", "", "", "", "", "LSD (0.10)  =", "5.7", "0.3", "ns", "37.8", "566.4"],
+    ['', '2012 BETTER VARIETIES Harvest Report for Minnesota Central  [ MNCE ]', '', '', '', '', '', '', '', '',
+     'ALL SEASON TEST'],
+    ['', 'Doug Toreen, Renville County, MN 55310          [ BIRD ISLAND ]', '', '', '', '', '', '', '', '',
+     '1.3 - 2.0 MAT. GROUP'],
+    ['PREV. CROP/HERB:', 'Corn / Surpass, Roundup', '', '', '', '', '', '', '', '', 'S2MNCE01'],
+    ['SOIL DESCRIPTION:', '', 'Canisteo clay loam, mod. well drained, non-irrigated', '', '', '', '', '', '', '', ''],
+    ['SOIL CONDITIONS:', '', 'High P, high K, 6.7 pH, 3.9% OM, Low SCN', '', '', '', '', '', '', '', '30" ROW SPACING'],
+    ['TILLAGE/CULTIVATION:', 'conventional w/ fall till', '', '', '', '', '', '', '', '', ''],
+    ['PEST MANAGEMENT:', 'Roundup twice', '', '', '', '', '', '', '', '', ''],
+    ['SEEDED - RATE:', 'May 15', '140 000 /A', '', '', '', '', '', '', 'TOP 30 for YIELD of 63 TESTED', ''],
+    ['HARVESTED - STAND:', 'Oct 3', '122 921 /A', '', '', '', '', '', '', 'AVERAGE of (3) REPLICATIONS', ''],
+    ['', '', '', '', 'SCN', 'Seed', 'Yield', 'Moisture', 'Lodging', 'Stand', 'Gross'],
+    ['Company/Brand', 'Product/Brand†', 'Technol.†', 'Mat.', 'Resist.', 'Trmt.†', 'Bu/A', '%', '%', '(x 1000)',
+     'Income'], ['Kruger', 'K2 1901', 'RR2Y', '1.9', 'R', 'Ac,PV', '56.4', '7.6', '0', '126.3', '$846'],
+    ['Stine', '19RA02 §', 'RR2Y', '1.9', 'R', 'CMB', '55.3', '7.6', '0', '120.0', '$830'],
+    ['Wensman', 'W 3190NR2', 'RR2Y', '1.9', 'R', 'Ac', '54.5', '7.6', '0', '119.5', '$818'],
+    ['Hefty', 'H17Y12', 'RR2Y', '1.7', 'MR', 'I', '53.7', '7.7', '0', '124.4', '$806'],
+    ['Dyna-Gro', 'S15RY53', 'RR2Y', '1.5', 'R', 'Ac', '53.6', '7.7', '0', '126.8', '$804'],
+    ['LG Seeds', 'C2050R2', 'RR2Y', '2.1', 'R', 'Ac', '53.6', '7.7', '0', '123.9', '$804'],
+    ['Titan Pro', '19M42', 'RR2Y', '1.9', 'R', 'CMB', '53.6', '7.7', '0', '121.0', '$804'],
+    ['Stine', '19RA02 (2) §', 'RR2Y', '1.9', 'R', 'CMB', '53.4', '7.7', '0', '123.9', '$801'],
+    ['Asgrow', 'AG1832 §', 'RR2Y', '1.8', 'MR', 'Ac,PV', '52.9', '7.7', '0', '122.0', '$794'],
+    ['Prairie Brand', 'PB-1566R2', 'RR2Y', '1.5', 'R', 'CMB', '52.8', '7.7', '0', '122.9', '$792'],
+    ['Channel', '1901R2', 'RR2Y', '1.9', 'R', 'Ac,PV', '52.8', '7.6', '0', '123.4', '$791'],
+    ['Titan Pro', '20M1', 'RR2Y', '2.0', 'R', 'Am', '52.5', '7.5', '0', '124.4', '$788'],
+    ['Kruger', 'K2-2002', 'RR2Y', '2.0', 'R', 'Ac,PV', '52.4', '7.9', '0', '125.4', '$786'],
+    ['Channel', '1700R2', 'RR2Y', '1.7', 'R', 'Ac,PV', '52.3', '7.9', '0', '123.9', '$784'],
+    ['Hefty', 'H16Y11', 'RR2Y', '1.6', 'MR', 'I', '51.4', '7.6', '0', '123.9', '$771'],
+    ['Anderson', '162R2Y', 'RR2Y', '1.6', 'R', 'None', '51.3', '7.5', '0', '119.5', '$770'],
+    ['Titan Pro', '15M22', 'RR2Y', '1.5', 'R', 'CMB', '51.3', '7.8', '0', '125.4', '$769'],
+    ['Dairyland', 'DSR-1710R2Y', 'RR2Y', '1.7', 'R', 'CMB', '51.3', '7.7', '0', '122.0', '$769'],
+    ['Hefty', 'H20R3', 'RR2Y', '2.0', 'MR', 'I', '50.5', '8.2', '0', '121.0', '$757'],
+    ['Prairie Brand', 'PB 1743R2', 'RR2Y', '1.7', 'R', 'CMB', '50.2', '7.7', '0', '125.8', '$752'],
+    ['Gold Country', '1741', 'RR2Y', '1.7', 'R', 'Ac', '50.1', '7.8', '0', '123.9', '$751'],
+    ['Trelay', '20RR43', 'RR2Y', '2.0', 'R', 'Ac,Ex', '49.9', '7.6', '0', '127.8', '$749'],
+    ['Hefty', 'H14R3', 'RR2Y', '1.4', 'MR', 'I', '49.7', '7.7', '0', '122.9', '$746'],
+    ['Prairie Brand', 'PB-2099NRR2', 'RR2Y', '2.0', 'R', 'CMB', '49.6', '7.8', '0', '126.3', '$743'],
+    ['Wensman', 'W 3174NR2', 'RR2Y', '1.7', 'R', 'Ac', '49.3', '7.6', '0', '122.5', '$740'],
+    ['Kruger', 'K2 1602', 'RR2Y', '1.6', 'R', 'Ac,PV', '48.7', '7.6', '0', '125.4', '$731'],
+    ['NK Brand', 'S18-C2 §', 'RR2Y', '1.8', 'R', 'CMB', '48.7', '7.7', '0', '126.8', '$731'],
+    ['Kruger', 'K2 1902', 'RR2Y', '1.9', 'R', 'Ac,PV', '48.7', '7.5', '0', '124.4', '$730'],
+    ['Prairie Brand', 'PB-1823R2', 'RR2Y', '1.8', 'R', 'None', '48.5', '7.6', '0', '121.0', '$727'],
+    ['Gold Country', '1541', 'RR2Y', '1.5', 'R', 'Ac', '48.4', '7.6', '0', '110.4', '$726'],
+    ['', '', '', '', '', 'Test Average  =', '47.6', '7.7', '0', '122.9', '$713'],
+    ['', '', '', '', '', 'LSD (0.10)  =', '5.7', '0.3', 'ns', '37.8', '566.4']
 ]