Bump version and update HISTORY.md

[MRG] Fix test data and drop python2 support

.deepsource.toml

@@ -0,0 +1,12 @@
+version = 1
+
+exclude_patterns = [
+  "camelot/ext/**"
+]
+
+[[analyzers]]
+name = "python"
+enabled = true
+
+[analyzers.meta]
+runtime_version = "3.x.x"

.github/FUNDING.yml

@@ -0,0 +1 @@
+open_collective: camelot

.gitignore

@@ -12,5 +12,8 @@ coverage.xml
 .pytest_cache/
 _build/
 
+.venv/
+htmlcov/
+
 # vscode
-.vscode
+.vscode

.travis.yml

@@ -8,14 +8,6 @@ install:
   - make install
 jobs:
   include:
-    - stage: test
-      script:
-        - make test
-      python: '2.7'
-    - stage: test
-      script:
-        - make test
-      python: '3.5'
     - stage: test
       script:
         - make test
@@ -25,8 +17,13 @@ jobs:
         - make test
       python: '3.7'
       dist: xenial
+    - stage: test
+      script:
+        - make test
+      python: '3.8'
+      dist: xenial
     - stage: coverage
-      python: '3.6'
+      python: '3.8'
       script:
         - make test
         - codecov --verbose

CONTRIBUTING.md

@@ -16,7 +16,7 @@ As the [Requests Code Of Conduct](http://docs.python-requests.org/en/master/dev/
 
 ## Your first contribution
 
-A great way to start contributing to Camelot is to pick an issue tagged with the [help wanted](https://github.com/socialcopsdev/camelot/labels/help%20wanted) tag or the [good first issue](https://github.com/socialcopsdev/camelot/labels/good%20first%20issue) tag. If you're unable to find a good first issue, feel free to contact the maintainer.
+A great way to start contributing to Camelot is to pick an issue tagged with the [help wanted](https://github.com/camelot-dev/camelot/labels/help%20wanted) tag or the [good first issue](https://github.com/camelot-dev/camelot/labels/good%20first%20issue) tag. If you're unable to find a good first issue, feel free to contact the maintainer.
 
 ## Setting up a development environment
 
@@ -36,7 +36,7 @@ $ pip install ".[dev]"
 
 ### Submit a pull request
 
-The preferred workflow for contributing to Camelot is to fork the [project repository](https://github.com/socialcopsdev/camelot) on GitHub, clone, develop on a branch and then finally submit a pull request. Here are the steps:
+The preferred workflow for contributing to Camelot is to fork the [project repository](https://github.com/camelot-dev/camelot) on GitHub, clone, develop on a branch and then finally submit a pull request. Here are the steps:
 
 1. Fork the project repository. Click on the ‘Fork’ button near the top of the page. This creates a copy of the code under your account on the GitHub.
 
@@ -106,7 +106,7 @@ The function docstrings are written using the [numpydoc](https://numpydoc.readth
 
 ## Filing Issues
 
-We use [GitHub issues](https://github.com/socialcopsdev/camelot/issues) to keep track of all issues and pull requests. Before opening an issue (which asks a question or reports a bug), please use GitHub search to look for existing issues (both open and closed) that may be similar.
+We use [GitHub issues](https://github.com/camelot-dev/camelot/issues) to keep track of all issues and pull requests. Before opening an issue (which asks a question or reports a bug), please use GitHub search to look for existing issues (both open and closed) that may be similar.
 
 ### Questions
 

HISTORY.md

@@ -4,6 +4,48 @@ Release History
 master
 ------
 
+0.8.0 (2020-05-24)
+------------------
+
+**Improvements**
+
+* Drop Python 2 support!
+    * Remove Python 2.7 and 3.5 support.
+    * Replace all instances of `.format` with f-strings.
+    * Remove all `__future__` imports.
+    * Fix HTTP 403 forbidden exception in read_pdf(url) and remove Python 2 urllib support.
+    * Fix test data.
+
+**Bugfixes**
+
+* Fix library discovery on Windows. [#32](https://github.com/camelot-dev/camelot/pull/32) by [KOLANICH](https://github.com/KOLANICH).
+* Fix calling convention of callback functions. [#34](https://github.com/camelot-dev/camelot/pull/34) by [KOLANICH](https://github.com/KOLANICH).
+
+0.7.3 (2019-07-07)
+------------------
+
+**Improvements**
+
+* Camelot now follows the Black code style! [#1](https://github.com/camelot-dev/camelot/pull/1) and [#3](https://github.com/camelot-dev/camelot/pull/3).
+
+**Bugfixes**
+
+* Fix Click.HelpFormatter monkey-patch. [#5](https://github.com/camelot-dev/camelot/pull/5) by [Dimiter Naydenov](https://github.com/dimitern).
+* Fix strip_text argument getting ignored. [#4](https://github.com/camelot-dev/camelot/pull/4) by [Dimiter Naydenov](https://github.com/dimitern).
+* [#25](https://github.com/camelot-dev/camelot/issues/25) edge_tol skipped in read_pdf. [#26](https://github.com/camelot-dev/camelot/pull/26) by Vinayak Mehta.
+* Fix pytest deprecation warning. [#2](https://github.com/camelot-dev/camelot/pull/2) by Vinayak Mehta.
+* [#293](https://github.com/socialcopsdev/camelot/issues/293) Split text ignores all text to the right of last cut. [#294](https://github.com/socialcopsdev/camelot/pull/294) by Vinayak Mehta.
+* [#277](https://github.com/socialcopsdev/camelot/issues/277) Sort TableList by order of tables in PDF. [#283](https://github.com/socialcopsdev/camelot/pull/283) by [Sym Roe](https://github.com/symroe).
+* [#312](https://github.com/socialcopsdev/camelot/issues/312) `table_regions` throws `ValueError` when `flavor='stream'`. [#332](https://github.com/socialcopsdev/camelot/pull/332) by Vinayak Mehta.
+
+0.7.2 (2019-01-10)
+------------------
+
+**Bugfixes**
+
+* [#245](https://github.com/socialcopsdev/camelot/issues/245) Fix AttributeError for encrypted files. [#251](https://github.com/socialcopsdev/camelot/pull/251) by Yatin Taluja.
+
+
 0.7.1 (2019-01-06)
 ------------------
 

LICENSE

@@ -1,5 +1,11 @@
 MIT License
 
+Modifications:
+
+Copyright (c) 2019 Camelot Developers
+
+Original project:
+
 Copyright (c) 2018 Peeply Private Ltd (Singapore)
 
 Permission is hereby granted, free of charge, to any person obtaining a copy

README.md

@@ -1,12 +1,14 @@
 <p align="center">
-   <img src="https://raw.githubusercontent.com/socialcopsdev/camelot/master/docs/_static/camelot.png" width="200">
+   <img src="https://raw.githubusercontent.com/camelot-dev/camelot/master/docs/_static/camelot.png" width="200">
 </p>
 
 # Camelot: PDF Table Extraction for Humans
 
-[![Build Status](https://travis-ci.org/socialcopsdev/camelot.svg?branch=master)](https://travis-ci.org/socialcopsdev/camelot) [![Documentation Status](https://readthedocs.org/projects/camelot-py/badge/?version=master)](https://camelot-py.readthedocs.io/en/master/)
- [![codecov.io](https://codecov.io/github/socialcopsdev/camelot/badge.svg?branch=master&service=github)](https://codecov.io/github/socialcopsdev/camelot?branch=master)
+[![Build Status](https://travis-ci.org/camelot-dev/camelot.svg?branch=master)](https://travis-ci.org/camelot-dev/camelot) [![Documentation Status](https://readthedocs.org/projects/camelot-py/badge/?version=master)](https://camelot-py.readthedocs.io/en/master/)
+ [![codecov.io](https://codecov.io/github/camelot-dev/camelot/badge.svg?branch=master&service=github)](https://codecov.io/github/camelot-dev/camelot?branch=master)
  [![image](https://img.shields.io/pypi/v/camelot-py.svg)](https://pypi.org/project/camelot-py/) [![image](https://img.shields.io/pypi/l/camelot-py.svg)](https://pypi.org/project/camelot-py/) [![image](https://img.shields.io/pypi/pyversions/camelot-py.svg)](https://pypi.org/project/camelot-py/) [![Gitter chat](https://badges.gitter.im/camelot-dev/Lobby.png)](https://gitter.im/camelot-dev/Lobby)
+[![image](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black) [![image](https://img.shields.io/badge/continous%20quality-deepsource-lightgrey)](https://deepsource.io/gh/camelot-dev/camelot/?ref=repository-badge)
+
 
 **Camelot** is a Python library that makes it easy for *anyone* to extract tables from PDF files!
 
@@ -14,7 +16,7 @@
 
 ---
 
-**Here's how you can extract tables from PDF files.** Check out the PDF used in this example [here](https://github.com/socialcopsdev/camelot/blob/master/docs/_static/pdf/foo.pdf).
+**Here's how you can extract tables from PDF files.** Check out the PDF used in this example [here](https://github.com/camelot-dev/camelot/blob/master/docs/_static/pdf/foo.pdf).
 
 <pre>
 >>> import camelot
@@ -55,7 +57,7 @@ There's a [command-line interface](https://camelot-py.readthedocs.io/en/master/u
 - Each table is a **pandas DataFrame**, which seamlessly integrates into [ETL and data analysis workflows](https://gist.github.com/vinayak-mehta/e5949f7c2410a0e12f25d3682dc9e873).
 - **Export** to multiple formats, including JSON, Excel, HTML and Sqlite.
 
-See [comparison with other PDF table extraction libraries and tools](https://github.com/socialcopsdev/camelot/wiki/Comparison-with-other-PDF-Table-Extraction-libraries-and-tools).
+See [comparison with other PDF table extraction libraries and tools](https://github.com/camelot-dev/camelot/wiki/Comparison-with-other-PDF-Table-Extraction-libraries-and-tools).
 
 ## Installation
 
@@ -69,7 +71,7 @@ $ conda install -c conda-forge camelot-py
 
 ### Using pip
 
-After [installing the dependencies](https://camelot-py.readthedocs.io/en/master/user/install-deps.html) ([tk](https://packages.ubuntu.com/trusty/python-tk) and [ghostscript](https://www.ghostscript.com/)), you can simply use pip to install Camelot:
+After [installing the dependencies](https://camelot-py.readthedocs.io/en/master/user/install-deps.html) ([tk](https://packages.ubuntu.com/bionic/python/python-tk) and [ghostscript](https://www.ghostscript.com/)), you can simply use pip to install Camelot:
 
 <pre>
 $ pip install camelot-py[cv]
@@ -80,7 +82,7 @@ $ pip install camelot-py[cv]
 After [installing the dependencies](https://camelot-py.readthedocs.io/en/master/user/install.html#using-pip), clone the repo using:
 
 <pre>
-$ git clone https://www.github.com/socialcopsdev/camelot
+$ git clone https://www.github.com/camelot-dev/camelot
 </pre>
 
 and install Camelot using pip:
@@ -103,7 +105,7 @@ The [Contributor's Guide](https://camelot-py.readthedocs.io/en/master/dev/contri
 You can check the latest sources with:
 
 <pre>
-$ git clone https://www.github.com/socialcopsdev/camelot
+$ git clone https://www.github.com/camelot-dev/camelot
 </pre>
 
 ### Setting up a development environment
@@ -122,10 +124,20 @@ After installation, you can run tests using:
 $ python setup.py test
 </pre>
 
+## Wrappers
+
+- [camelot-php](https://github.com/randomstate/camelot-php) provides a [PHP](https://www.php.net/) wrapper on Camelot.
+
 ## Versioning
 
-Camelot uses [Semantic Versioning](https://semver.org/). For the available versions, see the tags on this repository. For the changelog, you can check out [HISTORY.md](https://github.com/socialcopsdev/camelot/blob/master/HISTORY.md).
+Camelot uses [Semantic Versioning](https://semver.org/). For the available versions, see the tags on this repository. For the changelog, you can check out [HISTORY.md](https://github.com/camelot-dev/camelot/blob/master/HISTORY.md).
 
 ## License
 
-This project is licensed under the MIT License, see the [LICENSE](https://github.com/socialcopsdev/camelot/blob/master/LICENSE) file for details.
+This project is licensed under the MIT License, see the [LICENSE](https://github.com/camelot-dev/camelot/blob/master/LICENSE) file for details.
+
+## Support the development
+
+You can support our work on Camelot with a one-time or monthly donation [on OpenCollective](https://opencollective.com/camelot). Organizations who use camelot can also sponsor the project for an acknowledgement on [our documentation site](https://camelot-py.readthedocs.io/en/master/) and this README.
+
+Special thanks to all the users, organizations and contributors that support Camelot!

camelot/__init__.py

@@ -2,26 +2,16 @@
 
 import logging
 
-from click import HelpFormatter
-
 from .__version__ import __version__
 from .io import read_pdf
 from .plotting import PlotMethods
 
 
-def _write_usage(self, prog, args='', prefix='Usage: '):
-    return self._write_usage('camelot', args, prefix=prefix)
-
-
-# monkey patch click.HelpFormatter
-HelpFormatter._write_usage = HelpFormatter.write_usage
-HelpFormatter.write_usage = _write_usage
-
 # set up logging
-logger = logging.getLogger('camelot')
+logger = logging.getLogger("camelot")
 
-format_string = '%(asctime)s - %(levelname)s - %(message)s'
-formatter = logging.Formatter(format_string, datefmt='%Y-%m-%dT%H:%M:%S')
+format_string = "%(asctime)s - %(levelname)s - %(message)s"
+formatter = logging.Formatter(format_string, datefmt="%Y-%m-%dT%H:%M:%S")
 handler = logging.StreamHandler()
 handler.setFormatter(formatter)
 

camelot/__main__.py

@@ -1,9 +1,7 @@
 # -*- coding: utf-8 -*-
 
-from __future__ import absolute_import
 
-
-__all__ = ('main',)
+__all__ = ("main",)
 
 
 def main():

camelot/__version__.py

@@ -1,23 +1,23 @@
 # -*- coding: utf-8 -*-
 
-VERSION = (0, 7, 1)
-PRERELEASE = None # alpha, beta or rc
+VERSION = (0, 8, 0)
+PRERELEASE = None  # alpha, beta or rc
 REVISION = None
 
 
 def generate_version(version, prerelease=None, revision=None):
-    version_parts = ['.'.join(map(str, version))]
+    version_parts = [".".join(map(str, version))]
     if prerelease is not None:
-        version_parts.append('-{}'.format(prerelease))
+        version_parts.append(f"-{prerelease}")
     if revision is not None:
-        version_parts.append('.{}'.format(revision))
-    return ''.join(version_parts)
+        version_parts.append(f".{revision}")
+    return "".join(version_parts)
 
 
-__title__ = 'camelot-py'
-__description__ = 'PDF Table Extraction for Humans.'
-__url__ = 'http://camelot-py.readthedocs.io/'
+__title__ = "camelot-py"
+__description__ = "PDF Table Extraction for Humans."
+__url__ = "http://camelot-py.readthedocs.io/"
 __version__ = generate_version(VERSION, prerelease=PRERELEASE, revision=REVISION)
-__author__ = 'Vinayak Mehta'
-__author_email__ = 'vmehta94@gmail.com'
-__license__ = 'MIT License'
+__author__ = "Vinayak Mehta"
+__author_email__ = "vmehta94@gmail.com"
+__license__ = "MIT License"

camelot/cli.py

@@ -3,6 +3,7 @@
 import logging
 
 import click
+
 try:
     import matplotlib.pyplot as plt
 except ImportError:
@@ -13,7 +14,7 @@ else:
 from . import __version__, read_pdf, plot
 
 
-logger = logging.getLogger('camelot')
+logger = logging.getLogger("camelot")
 logger.setLevel(logging.INFO)
 
 
@@ -28,25 +29,49 @@ class Config(object):
 pass_config = click.make_pass_decorator(Config)
 
 
-@click.group()
+@click.group(name="camelot")
 @click.version_option(version=__version__)
-@click.option('-q', '--quiet', is_flag=False, help='Suppress logs and warnings.')
-@click.option('-p', '--pages', default='1', help='Comma-separated page numbers.'
-              ' Example: 1,3,4 or 1,4-end or all.')
-@click.option('-pw', '--password', help='Password for decryption.')
-@click.option('-o', '--output', help='Output file path.')
-@click.option('-f', '--format',
-              type=click.Choice(['csv', 'json', 'excel', 'html', 'sqlite']),
-              help='Output file format.')
-@click.option('-z', '--zip', is_flag=True, help='Create ZIP archive.')
-@click.option('-split', '--split_text', is_flag=True,
-              help='Split text that spans across multiple cells.')
-@click.option('-flag', '--flag_size', is_flag=True, help='Flag text based on'
-              ' font size. Useful to detect super/subscripts.')
-@click.option('-strip', '--strip_text', help='Characters that should be stripped from a string before'
-              ' assigning it to a cell.')
-@click.option('-M', '--margins', nargs=3, default=(1.0, 0.5, 0.1),
-              help='PDFMiner char_margin, line_margin and word_margin.')
+@click.option("-q", "--quiet", is_flag=False, help="Suppress logs and warnings.")
+@click.option(
+    "-p",
+    "--pages",
+    default="1",
+    help="Comma-separated page numbers." " Example: 1,3,4 or 1,4-end or all.",
+)
+@click.option("-pw", "--password", help="Password for decryption.")
+@click.option("-o", "--output", help="Output file path.")
+@click.option(
+    "-f",
+    "--format",
+    type=click.Choice(["csv", "json", "excel", "html", "sqlite"]),
+    help="Output file format.",
+)
+@click.option("-z", "--zip", is_flag=True, help="Create ZIP archive.")
+@click.option(
+    "-split",
+    "--split_text",
+    is_flag=True,
+    help="Split text that spans across multiple cells.",
+)
+@click.option(
+    "-flag",
+    "--flag_size",
+    is_flag=True,
+    help="Flag text based on" " font size. Useful to detect super/subscripts.",
+)
+@click.option(
+    "-strip",
+    "--strip_text",
+    help="Characters that should be stripped from a string before"
+    " assigning it to a cell.",
+)
+@click.option(
+    "-M",
+    "--margins",
+    nargs=3,
+    default=(1.0, 0.5, 0.1),
+    help="PDFMiner char_margin, line_margin and word_margin.",
+)
 @click.pass_context
 def cli(ctx, *args, **kwargs):
     """Camelot: PDF Table Extraction for Humans"""
@@ -55,79 +80,131 @@ def cli(ctx, *args, **kwargs):
         ctx.obj.set_config(key, value)
 
 
-@cli.command('lattice')
-@click.option('-R', '--table_regions', default=[], multiple=True,
-              help='Page regions to analyze. Example: x1,y1,x2,y2'
-              ' where x1, y1 -> left-top and x2, y2 -> right-bottom.')
-@click.option('-T', '--table_areas', default=[], multiple=True,
-              help='Table areas to process. Example: x1,y1,x2,y2'
-              ' where x1, y1 -> left-top and x2, y2 -> right-bottom.')
-@click.option('-back', '--process_background', is_flag=True,
-              help='Process background lines.')
-@click.option('-scale', '--line_scale', default=15,
-              help='Line size scaling factor. The larger the value,'
-              ' the smaller the detected lines.')
-@click.option('-copy', '--copy_text', default=[], type=click.Choice(['h', 'v']),
-              multiple=True, help='Direction in which text in a spanning cell'
-              ' will be copied over.')
-@click.option('-shift', '--shift_text', default=['l', 't'],
-              type=click.Choice(['', 'l', 'r', 't', 'b']), multiple=True,
-              help='Direction in which text in a spanning cell will flow.')
-@click.option('-l', '--line_tol', default=2,
-              help='Tolerance parameter used to merge close vertical'
-              ' and horizontal lines.')
-@click.option('-j', '--joint_tol', default=2,
-              help='Tolerance parameter used to decide whether'
-              ' the detected lines and points lie close to each other.')
-@click.option('-block', '--threshold_blocksize', default=15,
-              help='For adaptive thresholding, size of a pixel'
-              ' neighborhood that is used to calculate a threshold value for'
-              ' the pixel. Example: 3, 5, 7, and so on.')
-@click.option('-const', '--threshold_constant', default=-2,
-              help='For adaptive thresholding, constant subtracted'
-              ' from the mean or weighted mean. Normally, it is positive but'
-              ' may be zero or negative as well.')
-@click.option('-I', '--iterations', default=0,
-              help='Number of times for erosion/dilation will be applied.')
-@click.option('-res', '--resolution', default=300,
-              help='Resolution used for PDF to PNG conversion.')
-@click.option('-plot', '--plot_type',
-              type=click.Choice(['text', 'grid', 'contour', 'joint', 'line']),
-              help='Plot elements found on PDF page for visual debugging.')
-@click.argument('filepath', type=click.Path(exists=True))
+@cli.command("lattice")
+@click.option(
+    "-R",
+    "--table_regions",
+    default=[],
+    multiple=True,
+    help="Page regions to analyze. Example: x1,y1,x2,y2"
+    " where x1, y1 -> left-top and x2, y2 -> right-bottom.",
+)
+@click.option(
+    "-T",
+    "--table_areas",
+    default=[],
+    multiple=True,
+    help="Table areas to process. Example: x1,y1,x2,y2"
+    " where x1, y1 -> left-top and x2, y2 -> right-bottom.",
+)
+@click.option(
+    "-back", "--process_background", is_flag=True, help="Process background lines."
+)
+@click.option(
+    "-scale",
+    "--line_scale",
+    default=15,
+    help="Line size scaling factor. The larger the value,"
+    " the smaller the detected lines.",
+)
+@click.option(
+    "-copy",
+    "--copy_text",
+    default=[],
+    type=click.Choice(["h", "v"]),
+    multiple=True,
+    help="Direction in which text in a spanning cell" " will be copied over.",
+)
+@click.option(
+    "-shift",
+    "--shift_text",
+    default=["l", "t"],
+    type=click.Choice(["", "l", "r", "t", "b"]),
+    multiple=True,
+    help="Direction in which text in a spanning cell will flow.",
+)
+@click.option(
+    "-l",
+    "--line_tol",
+    default=2,
+    help="Tolerance parameter used to merge close vertical" " and horizontal lines.",
+)
+@click.option(
+    "-j",
+    "--joint_tol",
+    default=2,
+    help="Tolerance parameter used to decide whether"
+    " the detected lines and points lie close to each other.",
+)
+@click.option(
+    "-block",
+    "--threshold_blocksize",
+    default=15,
+    help="For adaptive thresholding, size of a pixel"
+    " neighborhood that is used to calculate a threshold value for"
+    " the pixel. Example: 3, 5, 7, and so on.",
+)
+@click.option(
+    "-const",
+    "--threshold_constant",
+    default=-2,
+    help="For adaptive thresholding, constant subtracted"
+    " from the mean or weighted mean. Normally, it is positive but"
+    " may be zero or negative as well.",
+)
+@click.option(
+    "-I",
+    "--iterations",
+    default=0,
+    help="Number of times for erosion/dilation will be applied.",
+)
+@click.option(
+    "-res",
+    "--resolution",
+    default=300,
+    help="Resolution used for PDF to PNG conversion.",
+)
+@click.option(
+    "-plot",
+    "--plot_type",
+    type=click.Choice(["text", "grid", "contour", "joint", "line"]),
+    help="Plot elements found on PDF page for visual debugging.",
+)
+@click.argument("filepath", type=click.Path(exists=True))
 @pass_config
 def lattice(c, *args, **kwargs):
     """Use lines between text to parse the table."""
     conf = c.config
-    pages = conf.pop('pages')
-    output = conf.pop('output')
-    f = conf.pop('format')
-    compress = conf.pop('zip')
-    quiet = conf.pop('quiet')
-    plot_type = kwargs.pop('plot_type')
-    filepath = kwargs.pop('filepath')
+    pages = conf.pop("pages")
+    output = conf.pop("output")
+    f = conf.pop("format")
+    compress = conf.pop("zip")
+    quiet = conf.pop("quiet")
+    plot_type = kwargs.pop("plot_type")
+    filepath = kwargs.pop("filepath")
     kwargs.update(conf)
 
-    table_regions = list(kwargs['table_regions'])
-    kwargs['table_regions'] = None if not table_regions else table_regions
-    table_areas = list(kwargs['table_areas'])
-    kwargs['table_areas'] = None if not table_areas else table_areas
-    copy_text = list(kwargs['copy_text'])
-    kwargs['copy_text'] = None if not copy_text else copy_text
-    kwargs['shift_text'] = list(kwargs['shift_text'])
+    table_regions = list(kwargs["table_regions"])
+    kwargs["table_regions"] = None if not table_regions else table_regions
+    table_areas = list(kwargs["table_areas"])
+    kwargs["table_areas"] = None if not table_areas else table_areas
+    copy_text = list(kwargs["copy_text"])
+    kwargs["copy_text"] = None if not copy_text else copy_text
+    kwargs["shift_text"] = list(kwargs["shift_text"])
 
     if plot_type is not None:
         if not _HAS_MPL:
-            raise ImportError('matplotlib is required for plotting.')
+            raise ImportError("matplotlib is required for plotting.")
     else:
         if output is None:
-            raise click.UsageError('Please specify output file path using --output')
+            raise click.UsageError("Please specify output file path using --output")
         if f is None:
-            raise click.UsageError('Please specify output file format using --format')
+            raise click.UsageError("Please specify output file format using --format")
 
-    tables = read_pdf(filepath, pages=pages, flavor='lattice',
-                      suppress_stdout=quiet, **kwargs)
-    click.echo('Found {} tables'.format(tables.n))
+    tables = read_pdf(
+        filepath, pages=pages, flavor="lattice", suppress_stdout=quiet, **kwargs
+    )
+    click.echo(f"Found {tables.n} tables")
     if plot_type is not None:
         for table in tables:
             plot(table, kind=plot_type)
@@ -136,57 +213,89 @@ def lattice(c, *args, **kwargs):
         tables.export(output, f=f, compress=compress)
 
 
-@cli.command('stream')
-@click.option('-R', '--table_regions', default=[], multiple=True,
-              help='Page regions to analyze. Example: x1,y1,x2,y2'
-              ' where x1, y1 -> left-top and x2, y2 -> right-bottom.')
-@click.option('-T', '--table_areas', default=[], multiple=True,
-              help='Table areas to process. Example: x1,y1,x2,y2'
-              ' where x1, y1 -> left-top and x2, y2 -> right-bottom.')
-@click.option('-C', '--columns', default=[], multiple=True,
-              help='X coordinates of column separators.')
-@click.option('-e', '--edge_tol', default=50, help='Tolerance parameter'
-              ' for extending textedges vertically.')
-@click.option('-r', '--row_tol', default=2, help='Tolerance parameter'
-              ' used to combine text vertically, to generate rows.')
-@click.option('-c', '--column_tol', default=0, help='Tolerance parameter'
-              ' used to combine text horizontally, to generate columns.')
-@click.option('-plot', '--plot_type',
-              type=click.Choice(['text', 'grid', 'contour', 'textedge']),
-              help='Plot elements found on PDF page for visual debugging.')
-@click.argument('filepath', type=click.Path(exists=True))
+@cli.command("stream")
+@click.option(
+    "-R",
+    "--table_regions",
+    default=[],
+    multiple=True,
+    help="Page regions to analyze. Example: x1,y1,x2,y2"
+    " where x1, y1 -> left-top and x2, y2 -> right-bottom.",
+)
+@click.option(
+    "-T",
+    "--table_areas",
+    default=[],
+    multiple=True,
+    help="Table areas to process. Example: x1,y1,x2,y2"
+    " where x1, y1 -> left-top and x2, y2 -> right-bottom.",
+)
+@click.option(
+    "-C",
+    "--columns",
+    default=[],
+    multiple=True,
+    help="X coordinates of column separators.",
+)
+@click.option(
+    "-e",
+    "--edge_tol",
+    default=50,
+    help="Tolerance parameter" " for extending textedges vertically.",
+)
+@click.option(
+    "-r",
+    "--row_tol",
+    default=2,
+    help="Tolerance parameter" " used to combine text vertically, to generate rows.",
+)
+@click.option(
+    "-c",
+    "--column_tol",
+    default=0,
+    help="Tolerance parameter"
+    " used to combine text horizontally, to generate columns.",
+)
+@click.option(
+    "-plot",
+    "--plot_type",
+    type=click.Choice(["text", "grid", "contour", "textedge"]),
+    help="Plot elements found on PDF page for visual debugging.",
+)
+@click.argument("filepath", type=click.Path(exists=True))
 @pass_config
 def stream(c, *args, **kwargs):
     """Use spaces between text to parse the table."""
     conf = c.config
-    pages = conf.pop('pages')
-    output = conf.pop('output')
-    f = conf.pop('format')
-    compress = conf.pop('zip')
-    quiet = conf.pop('quiet')
-    plot_type = kwargs.pop('plot_type')
-    filepath = kwargs.pop('filepath')
+    pages = conf.pop("pages")
+    output = conf.pop("output")
+    f = conf.pop("format")
+    compress = conf.pop("zip")
+    quiet = conf.pop("quiet")
+    plot_type = kwargs.pop("plot_type")
+    filepath = kwargs.pop("filepath")
     kwargs.update(conf)
 
-    table_regions = list(kwargs['table_regions'])
-    kwargs['table_regions'] = None if not table_regions else table_regions
-    table_areas = list(kwargs['table_areas'])
-    kwargs['table_areas'] = None if not table_areas else table_areas
-    columns = list(kwargs['columns'])
-    kwargs['columns'] = None if not columns else columns
+    table_regions = list(kwargs["table_regions"])
+    kwargs["table_regions"] = None if not table_regions else table_regions
+    table_areas = list(kwargs["table_areas"])
+    kwargs["table_areas"] = None if not table_areas else table_areas
+    columns = list(kwargs["columns"])
+    kwargs["columns"] = None if not columns else columns
 
     if plot_type is not None:
         if not _HAS_MPL:
-            raise ImportError('matplotlib is required for plotting.')
+            raise ImportError("matplotlib is required for plotting.")
     else:
         if output is None:
-            raise click.UsageError('Please specify output file path using --output')
+            raise click.UsageError("Please specify output file path using --output")
         if f is None:
-            raise click.UsageError('Please specify output file format using --format')
+            raise click.UsageError("Please specify output file format using --format")
 
-    tables = read_pdf(filepath, pages=pages, flavor='stream',
-                      suppress_stdout=quiet, **kwargs)
-    click.echo('Found {} tables'.format(tables.n))
+    tables = read_pdf(
+        filepath, pages=pages, flavor="stream", suppress_stdout=quiet, **kwargs
+    )
+    click.echo(f"Found {tables.n} tables")
     if plot_type is not None:
         for table in tables:
             plot(table, kind=plot_type)

camelot/core.py

@@ -42,7 +42,8 @@ class TextEdge(object):
         TEXTEDGE_REQUIRED_ELEMENTS horizontal text rows.
 
     """
-    def __init__(self, x, y0, y1, align='left'):
+
+    def __init__(self, x, y0, y1, align="left"):
         self.x = x
         self.y0 = y0
         self.y1 = y1
@@ -51,8 +52,10 @@ class TextEdge(object):
         self.is_valid = False
 
     def __repr__(self):
-        return '<TextEdge x={} y0={} y1={} align={} valid={}>'.format(
-            round(self.x, 2), round(self.y0, 2), round(self.y1, 2), self.align, self.is_valid)
+        x = round(self.x, 2)
+        y0 = round(self.y0, 2)
+        y1 = round(self.y1, 2)
+        return f"<TextEdge x={x} y0={y0} y1={y1} align={self.align} valid={self.is_valid}>"
 
     def update_coords(self, x, y0, edge_tol=50):
         """Updates the text edge's x and bottom y coordinates and sets
@@ -73,9 +76,10 @@ class TextEdges(object):
     the PDF page. The dict has three keys based on the alignments,
     and each key's value is a list of camelot.core.TextEdge objects.
     """
+
     def __init__(self, edge_tol=50):
         self.edge_tol = edge_tol
-        self._textedges = {'left': [], 'right': [], 'middle': []}
+        self._textedges = {"left": [], "right": [], "middle": []}
 
     @staticmethod
     def get_x_coord(textline, align):
@@ -85,7 +89,7 @@ class TextEdges(object):
         x_left = textline.x0
         x_right = textline.x1
         x_middle = x_left + (x_right - x_left) / 2.0
-        x_coord = {'left': x_left, 'middle': x_middle, 'right': x_right}
+        x_coord = {"left": x_left, "middle": x_middle, "right": x_right}
         return x_coord[align]
 
     def find(self, x_coord, align):
@@ -109,21 +113,22 @@ class TextEdges(object):
     def update(self, textline):
         """Updates an existing text edge in the current dict.
         """
-        for align in ['left', 'right', 'middle']:
+        for align in ["left", "right", "middle"]:
             x_coord = self.get_x_coord(textline, align)
             idx = self.find(x_coord, align)
             if idx is None:
                 self.add(textline, align)
             else:
                 self._textedges[align][idx].update_coords(
-                    x_coord, textline.y0, edge_tol=self.edge_tol)
+                    x_coord, textline.y0, edge_tol=self.edge_tol
+                )
 
     def generate(self, textlines):
         """Generates the text edges dict based on horizontal text
         rows.
         """
         for tl in textlines:
-            if len(tl.get_text().strip()) > 1: # TODO: hacky
+            if len(tl.get_text().strip()) > 1:  # TODO: hacky
                 self.update(tl)
 
     def get_relevant(self):
@@ -132,9 +137,15 @@ class TextEdges(object):
         the most.
         """
         intersections_sum = {
-            'left': sum(te.intersections for te in self._textedges['left'] if te.is_valid),
-            'right': sum(te.intersections for te in self._textedges['right'] if te.is_valid),
-            'middle': sum(te.intersections for te in self._textedges['middle'] if te.is_valid)
+            "left": sum(
+                te.intersections for te in self._textedges["left"] if te.is_valid
+            ),
+            "right": sum(
+                te.intersections for te in self._textedges["right"] if te.is_valid
+            ),
+            "middle": sum(
+                te.intersections for te in self._textedges["middle"] if te.is_valid
+            ),
         }
 
         # TODO: naive
@@ -147,6 +158,7 @@ class TextEdges(object):
         """Returns a dict of interesting table areas on the PDF page
         calculated using relevant text edges.
         """
+
         def pad(area, average_row_height):
             x0 = area[0] - TABLE_AREA_PADDING
             y0 = area[1] - TABLE_AREA_PADDING
@@ -175,7 +187,11 @@ class TextEdges(object):
                     else:
                         table_areas.pop(found)
                         updated_area = (
-                            found[0], min(te.y0, found[1]), max(found[2], te.x), max(found[3], te.y1))
+                            found[0],
+                            min(te.y0, found[1]),
+                            max(found[2], te.x),
+                            max(found[3], te.y1),
+                        )
                         table_areas[updated_area] = None
 
         # extend table areas based on textlines that overlap
@@ -196,7 +212,11 @@ class TextEdges(object):
             if found is not None:
                 table_areas.pop(found)
                 updated_area = (
-                    min(tl.x0, found[0]), min(tl.y0, found[1]), max(found[2], tl.x1), max(found[3], tl.y1))
+                    min(tl.x0, found[0]),
+                    min(tl.y0, found[1]),
+                    max(found[2], tl.x1),
+                    max(found[3], tl.y1),
+                )
                 table_areas[updated_area] = None
         average_textline_height = sum_textline_height / float(len(textlines))
 
@@ -265,11 +285,14 @@ class Cell(object):
         self.bottom = False
         self.hspan = False
         self.vspan = False
-        self._text = ''
+        self._text = ""
 
     def __repr__(self):
-        return '<Cell x1={} y1={} x2={} y2={}>'.format(
-            round(self.x1, 2), round(self.y1, 2), round(self.x2, 2), round(self.y2, 2))
+        x1 = round(self.x1, 2)
+        y1 = round(self.y1, 2)
+        x2 = round(self.x2, 2)
+        y2 = round(self.y2, 2)
+        return f"<Cell x1={x1} y1={y1} x2={x2} y2={y2}>"
 
     @property
     def text(self):
@@ -277,7 +300,7 @@ class Cell(object):
 
     @text.setter
     def text(self, t):
-        self._text = ''.join([self._text, t])
+        self._text = "".join([self._text, t])
 
     @property
     def bound(self):
@@ -314,11 +337,11 @@ class Table(object):
         PDF page number.
 
     """
+
     def __init__(self, cols, rows):
         self.cols = cols
         self.rows = rows
-        self.cells = [[Cell(c[0], r[1], c[1], r[0])
-                       for c in cols] for r in rows]
+        self.cells = [[Cell(c[0], r[1], c[1], r[0]) for c in cols] for r in rows]
         self.df = None
         self.shape = (0, 0)
         self.accuracy = 0
@@ -327,7 +350,14 @@ class Table(object):
         self.page = None
 
     def __repr__(self):
-        return '<{} shape={}>'.format(self.__class__.__name__, self.shape)
+        return f"<{self.__class__.__name__} shape={self.shape}>"
+
+    def __lt__(self, other):
+        if self.page == other.page:
+            if self.order < other.order:
+                return True
+        if self.page < other.page:
+            return True
 
     @property
     def data(self):
@@ -345,10 +375,10 @@ class Table(object):
         """
         # pretty?
         report = {
-            'accuracy': round(self.accuracy, 2),
-            'whitespace': round(self.whitespace, 2),
-            'order': self.order,
-            'page': self.page
+            "accuracy": round(self.accuracy, 2),
+            "whitespace": round(self.whitespace, 2),
+            "order": self.order,
+            "page": self.page,
         }
         return report
 
@@ -376,12 +406,21 @@ class Table(object):
         for v in vertical:
             # find closest x coord
             # iterate over y coords and find closest start and end points
-            i = [i for i, t in enumerate(self.cols)
-                 if np.isclose(v[0], t[0], atol=joint_tol)]
-            j = [j for j, t in enumerate(self.rows)
-                 if np.isclose(v[3], t[0], atol=joint_tol)]
-            k = [k for k, t in enumerate(self.rows)
-                 if np.isclose(v[1], t[0], atol=joint_tol)]
+            i = [
+                i
+                for i, t in enumerate(self.cols)
+                if np.isclose(v[0], t[0], atol=joint_tol)
+            ]
+            j = [
+                j
+                for j, t in enumerate(self.rows)
+                if np.isclose(v[3], t[0], atol=joint_tol)
+            ]
+            k = [
+                k
+                for k, t in enumerate(self.rows)
+                if np.isclose(v[1], t[0], atol=joint_tol)
+            ]
             if not j:
                 continue
             J = j[0]
@@ -427,12 +466,21 @@ class Table(object):
         for h in horizontal:
             # find closest y coord
             # iterate over x coords and find closest start and end points
-            i = [i for i, t in enumerate(self.rows)
-                 if np.isclose(h[1], t[0], atol=joint_tol)]
-            j = [j for j, t in enumerate(self.cols)
-                 if np.isclose(h[0], t[0], atol=joint_tol)]
-            k = [k for k, t in enumerate(self.cols)
-                 if np.isclose(h[2], t[0], atol=joint_tol)]
+            i = [
+                i
+                for i, t in enumerate(self.rows)
+                if np.isclose(h[1], t[0], atol=joint_tol)
+            ]
+            j = [
+                j
+                for j, t in enumerate(self.cols)
+                if np.isclose(h[0], t[0], atol=joint_tol)
+            ]
+            k = [
+                k
+                for k, t in enumerate(self.cols)
+                if np.isclose(h[2], t[0], atol=joint_tol)
+            ]
             if not j:
                 continue
             J = j[0]
@@ -530,12 +578,7 @@ class Table(object):
             Output filepath.
 
         """
-        kw = {
-            'encoding': 'utf-8',
-            'index': False,
-            'header': False,
-            'quoting': 1
-        }
+        kw = {"encoding": "utf-8", "index": False, "header": False, "quoting": 1}
         kw.update(kwargs)
         self.df.to_csv(path, **kw)
 
@@ -550,12 +593,10 @@ class Table(object):
             Output filepath.
 
         """
-        kw = {
-            'orient': 'records'
-        }
+        kw = {"orient": "records"}
         kw.update(kwargs)
         json_string = self.df.to_json(**kw)
-        with open(path, 'w') as f:
+        with open(path, "w") as f:
             f.write(json_string)
 
     def to_excel(self, path, **kwargs):
@@ -570,8 +611,8 @@ class Table(object):
 
         """
         kw = {
-            'sheet_name': 'page-{}-table-{}'.format(self.page, self.order),
-            'encoding': 'utf-8'
+            "sheet_name": f"page-{self.page}-table-{self.order}",
+            "encoding": "utf-8",
         }
         kw.update(kwargs)
         writer = pd.ExcelWriter(path)
@@ -590,7 +631,7 @@ class Table(object):
 
         """
         html_string = self.df.to_html(**kwargs)
-        with open(path, 'w') as f:
+        with open(path, "w") as f:
             f.write(html_string)
 
     def to_sqlite(self, path, **kwargs):
@@ -604,13 +645,10 @@ class Table(object):
             Output filepath.
 
         """
-        kw = {
-            'if_exists': 'replace',
-            'index': False
-        }
+        kw = {"if_exists": "replace", "index": False}
         kw.update(kwargs)
         conn = sqlite3.connect(path)
-        table_name = 'page-{}-table-{}'.format(self.page, self.order)
+        table_name = f"page-{self.page}-table-{self.order}"
         self.df.to_sql(table_name, conn, **kw)
         conn.commit()
         conn.close()
@@ -626,12 +664,12 @@ class TableList(object):
         Number of tables in the list.
 
     """
+
     def __init__(self, tables):
         self._tables = tables
 
     def __repr__(self):
-        return '<{} n={}>'.format(
-            self.__class__.__name__, self.n)
+        return f"<{self.__class__.__name__} n={self.n}>"
 
     def __len__(self):
         return len(self._tables)
@@ -641,37 +679,35 @@ class TableList(object):
 
     @staticmethod
     def _format_func(table, f):
-        return getattr(table, 'to_{}'.format(f))
+        return getattr(table, f"to_{f}")
 
     @property
     def n(self):
         return len(self)
 
     def _write_file(self, f=None, **kwargs):
-        dirname = kwargs.get('dirname')
-        root = kwargs.get('root')
-        ext = kwargs.get('ext')
+        dirname = kwargs.get("dirname")
+        root = kwargs.get("root")
+        ext = kwargs.get("ext")
         for table in self._tables:
-            filename = os.path.join('{}-page-{}-table-{}{}'.format(
-                                    root, table.page, table.order, ext))
+            filename = f"{root}-page-{table.page}-table-{table.order}{ext}"
             filepath = os.path.join(dirname, filename)
             to_format = self._format_func(table, f)
             to_format(filepath)
 
     def _compress_dir(self, **kwargs):
-        path = kwargs.get('path')
-        dirname = kwargs.get('dirname')
-        root = kwargs.get('root')
-        ext = kwargs.get('ext')
-        zipname = os.path.join(os.path.dirname(path), root) + '.zip'
-        with zipfile.ZipFile(zipname, 'w', allowZip64=True) as z:
+        path = kwargs.get("path")
+        dirname = kwargs.get("dirname")
+        root = kwargs.get("root")
+        ext = kwargs.get("ext")
+        zipname = os.path.join(os.path.dirname(path), root) + ".zip"
+        with zipfile.ZipFile(zipname, "w", allowZip64=True) as z:
             for table in self._tables:
-                filename = os.path.join('{}-page-{}-table-{}{}'.format(
-                                        root, table.page, table.order, ext))
+                filename = f"{root}-page-{table.page}-table-{table.order}{ext}"
                 filepath = os.path.join(dirname, filename)
                 z.write(filepath, os.path.basename(filepath))
 
-    def export(self, path, f='csv', compress=False):
+    def export(self, path, f="csv", compress=False):
         """Exports the list of tables to specified file format.
 
         Parameters
@@ -690,33 +726,28 @@ class TableList(object):
         if compress:
             dirname = tempfile.mkdtemp()
 
-        kwargs = {
-            'path': path,
-            'dirname': dirname,
-            'root': root,
-            'ext': ext
-        }
+        kwargs = {"path": path, "dirname": dirname, "root": root, "ext": ext}
 
-        if f in ['csv', 'json', 'html']:
+        if f in ["csv", "json", "html"]:
             self._write_file(f=f, **kwargs)
             if compress:
                 self._compress_dir(**kwargs)
-        elif f == 'excel':
+        elif f == "excel":
             filepath = os.path.join(dirname, basename)
             writer = pd.ExcelWriter(filepath)
             for table in self._tables:
-                sheet_name = 'page-{}-table-{}'.format(table.page, table.order)
-                table.df.to_excel(writer, sheet_name=sheet_name, encoding='utf-8')
+                sheet_name = f"page-{table.page}-table-{table.order}"
+                table.df.to_excel(writer, sheet_name=sheet_name, encoding="utf-8")
             writer.save()
             if compress:
-                zipname = os.path.join(os.path.dirname(path), root) + '.zip'
-                with zipfile.ZipFile(zipname, 'w', allowZip64=True) as z:
+                zipname = os.path.join(os.path.dirname(path), root) + ".zip"
+                with zipfile.ZipFile(zipname, "w", allowZip64=True) as z:
                     z.write(filepath, os.path.basename(filepath))
-        elif f == 'sqlite':
+        elif f == "sqlite":
             filepath = os.path.join(dirname, basename)
             for table in self._tables:
                 table.to_sqlite(filepath)
             if compress:
-                zipname = os.path.join(os.path.dirname(path), root) + '.zip'
-                with zipfile.ZipFile(zipname, 'w', allowZip64=True) as z:
+                zipname = os.path.join(os.path.dirname(path), root) + ".zip"
+                with zipfile.ZipFile(zipname, "w", allowZip64=True) as z:
                     z.write(filepath, os.path.basename(filepath))

camelot/ext/ghostscript/__init__.py

@@ -24,10 +24,10 @@ ghostscript - A Python interface for the Ghostscript interpreter C-API
 from . import _gsprint as gs
 
 
-__author__ = 'Hartmut Goebel <h.goebel@crazy-compilers.com>'
-__copyright__ = 'Copyright 2010-2018 by Hartmut Goebel <h.goebel@crazy-compilers.com>'
-__license__ = 'GNU General Public License version 3 (GPL v3)'
-__version__ = '0.6'
+__author__ = "Hartmut Goebel <h.goebel@crazy-compilers.com>"
+__copyright__ = "Copyright 2010-2018 by Hartmut Goebel <h.goebel@crazy-compilers.com>"
+__license__ = "GNU General Public License version 3 (GPL v3)"
+__version__ = "0.6"
 
 
 class __Ghostscript(object):
@@ -87,10 +87,13 @@ def Ghostscript(*args, **kwargs):
     # Ghostscript only supports a single instance
     if __instance__ is None:
         __instance__ = gs.new_instance()
-    return __Ghostscript(__instance__, args,
-                         stdin=kwargs.get('stdin', None),
-                         stdout=kwargs.get('stdout', None),
-                         stderr=kwargs.get('stderr', None))
+    return __Ghostscript(
+        __instance__,
+        args,
+        stdin=kwargs.get("stdin", None),
+        stdout=kwargs.get("stdout", None),
+        stderr=kwargs.get("stderr", None),
+    )
 
 
 __instance__ = None

camelot/ext/ghostscript/_gsprint.py

@@ -42,10 +42,10 @@ e_Info = -110
 #
 e_Quit = -101
 
-__author__ = 'Hartmut Goebel <h.goebel@crazy-compilers.com>'
-__copyright__ = 'Copyright 2010-2018 by Hartmut Goebel <h.goebel@crazy-compilers.com>'
-__license__ = 'GNU General Public License version 3 (GPL v3)'
-__version__ = '0.6'
+__author__ = "Hartmut Goebel <h.goebel@crazy-compilers.com>"
+__copyright__ = "Copyright 2010-2018 by Hartmut Goebel <h.goebel@crazy-compilers.com>"
+__license__ = "GNU General Public License version 3 (GPL v3)"
+__version__ = "0.6"
 
 gs_main_instance = c_void_p
 display_callback = c_void_p
@@ -55,7 +55,7 @@ display_callback = c_void_p
 
 class GhostscriptError(Exception):
     def __init__(self, ecode):
-         self.code = ecode
+        self.code = ecode
 
 
 def new_instance():
@@ -82,13 +82,17 @@ def delete_instance(instance):
     return libgs.gsapi_delete_instance(instance)
 
 
-c_stdstream_call_t = CFUNCTYPE(c_int, gs_main_instance, POINTER(c_char), c_int)
+if sys.platform == "win32":
+    c_stdstream_call_t = WINFUNCTYPE(c_int, gs_main_instance, POINTER(c_char), c_int)
+else:
+    c_stdstream_call_t = CFUNCTYPE(c_int, gs_main_instance, POINTER(c_char), c_int)
 
 
 def _wrap_stdin(infp):
     """Wrap a filehandle into a C function to be used as `stdin` callback
     for ``set_stdio``. The filehandle has to support the readline() method.
     """
+
     def _wrap(instance, dest, count):
         try:
             data = infp.readline(count)
@@ -110,6 +114,7 @@ def _wrap_stdout(outfp):
     `stderr` callback for ``set_stdio``. The filehandle has to support the
     write() and flush() methods.
     """
+
     def _wrap(instance, str, count):
         outfp.write(str[:count])
         outfp.flush()
@@ -187,11 +192,23 @@ def __win32_finddll():
         import winreg
     except ImportError:
         # assume Python 2
-        from _winreg import OpenKey, CloseKey, EnumKey, QueryValueEx, \
-            QueryInfoKey, HKEY_LOCAL_MACHINE
+        from _winreg import (
+            OpenKey,
+            CloseKey,
+            EnumKey,
+            QueryValueEx,
+            QueryInfoKey,
+            HKEY_LOCAL_MACHINE,
+        )
     else:
-        from winreg import OpenKey, CloseKey, EnumKey, QueryValueEx, \
-            QueryInfoKey, HKEY_LOCAL_MACHINE
+        from winreg import (
+            OpenKey,
+            CloseKey,
+            EnumKey,
+            QueryValueEx,
+            QueryInfoKey,
+            HKEY_LOCAL_MACHINE,
+        )
 
     from distutils.version import LooseVersion
     import os
@@ -199,15 +216,19 @@ def __win32_finddll():
     dlls = []
     # Look up different variants of Ghostscript and take the highest
     # version for which the DLL is to be found in the filesystem.
-    for key_name in ('AFPL Ghostscript', 'Aladdin Ghostscript',
-                     'GNU Ghostscript', 'GPL Ghostscript'):
+    for key_name in (
+        "AFPL Ghostscript",
+        "Aladdin Ghostscript",
+        "GNU Ghostscript",
+        "GPL Ghostscript",
+    ):
         try:
             k1 = OpenKey(HKEY_LOCAL_MACHINE, "Software\\%s" % key_name)
             for num in range(0, QueryInfoKey(k1)[0]):
                 version = EnumKey(k1, num)
                 try:
                     k2 = OpenKey(k1, version)
-                    dll_path = QueryValueEx(k2, 'GS_DLL')[0]
+                    dll_path = QueryValueEx(k2, "GS_DLL")[0]
                     CloseKey(k2)
                     if os.path.exists(dll_path):
                         dlls.append((LooseVersion(version), dll_path))
@@ -223,21 +244,27 @@ def __win32_finddll():
         return None
 
 
-if sys.platform == 'win32':
+if sys.platform == "win32":
     libgs = __win32_finddll()
     if not libgs:
-        raise RuntimeError('Please make sure that Ghostscript is installed')
+        import ctypes.util
+
+        libgs = ctypes.util.find_library(
+            "".join(("gsdll", str(ctypes.sizeof(ctypes.c_voidp) * 8), ".dll"))
+        )  # finds in %PATH%
+    if not libgs:
+        raise RuntimeError("Please make sure that Ghostscript is installed")
     libgs = windll.LoadLibrary(libgs)
 else:
     try:
-        libgs = cdll.LoadLibrary('libgs.so')
+        libgs = cdll.LoadLibrary("libgs.so")
     except OSError:
         # shared object file not found
         import ctypes.util
 
-        libgs = ctypes.util.find_library('gs')
+        libgs = ctypes.util.find_library("gs")
         if not libgs:
-            raise RuntimeError('Please make sure that Ghostscript is installed')
+            raise RuntimeError("Please make sure that Ghostscript is installed")
         libgs = cdll.LoadLibrary(libgs)
 
 del __win32_finddll

camelot/handlers.py

@@ -7,8 +7,14 @@ from PyPDF2 import PdfFileReader, PdfFileWriter
 
 from .core import TableList
 from .parsers import Stream, Lattice
-from .utils import (TemporaryDirectory, get_page_layout, get_text_objects,
-                    get_rotation, is_url, download_url)
+from .utils import (
+    TemporaryDirectory,
+    get_page_layout,
+    get_text_objects,
+    get_rotation,
+    is_url,
+    download_url,
+)
 
 
 class PDFHandler(object):
@@ -27,19 +33,21 @@ class PDFHandler(object):
         Password for decryption.
 
     """
-    def __init__(self, filepath, pages='1', password=None):
+
+    def __init__(self, filepath, pages="1", password=None):
         if is_url(filepath):
             filepath = download_url(filepath)
         self.filepath = filepath
-        if not filepath.lower().endswith('.pdf'):
+        if not filepath.lower().endswith(".pdf"):
             raise NotImplementedError("File format not supported")
-        self.pages = self._get_pages(self.filepath, pages)
+
         if password is None:
-            self.password = ''
+            self.password = ""
         else:
             self.password = password
             if sys.version_info[0] < 3:
-                self.password = self.password.encode('ascii')
+                self.password = self.password.encode("ascii")
+        self.pages = self._get_pages(self.filepath, pages)
 
     def _get_pages(self, filepath, pages):
         """Converts pages string to list of ints.
@@ -59,26 +67,26 @@ class PDFHandler(object):
 
         """
         page_numbers = []
-        if pages == '1':
-            page_numbers.append({'start': 1, 'end': 1})
+        if pages == "1":
+            page_numbers.append({"start": 1, "end": 1})
         else:
-            infile = PdfFileReader(open(filepath, 'rb'), strict=False)
+            infile = PdfFileReader(open(filepath, "rb"), strict=False)
             if infile.isEncrypted:
                 infile.decrypt(self.password)
-            if pages == 'all':
-                page_numbers.append({'start': 1, 'end': infile.getNumPages()})
+            if pages == "all":
+                page_numbers.append({"start": 1, "end": infile.getNumPages()})
             else:
-                for r in pages.split(','):
-                    if '-' in r:
-                        a, b = r.split('-')
-                        if b == 'end':
+                for r in pages.split(","):
+                    if "-" in r:
+                        a, b = r.split("-")
+                        if b == "end":
                             b = infile.getNumPages()
-                        page_numbers.append({'start': int(a), 'end': int(b)})
+                        page_numbers.append({"start": int(a), "end": int(b)})
                     else:
-                        page_numbers.append({'start': int(r), 'end': int(r)})
+                        page_numbers.append({"start": int(r), "end": int(r)})
         P = []
         for p in page_numbers:
-            P.extend(range(p['start'], p['end'] + 1))
+            P.extend(range(p["start"], p["end"] + 1))
         return sorted(set(P))
 
     def _save_page(self, filepath, page, temp):
@@ -94,16 +102,16 @@ class PDFHandler(object):
             Tmp directory.
 
         """
-        with open(filepath, 'rb') as fileobj:
+        with open(filepath, "rb") as fileobj:
             infile = PdfFileReader(fileobj, strict=False)
             if infile.isEncrypted:
                 infile.decrypt(self.password)
-            fpath = os.path.join(temp, 'page-{0}.pdf'.format(page))
+            fpath = os.path.join(temp, f"page-{page}.pdf")
             froot, fext = os.path.splitext(fpath)
             p = infile.getPage(page - 1)
             outfile = PdfFileWriter()
             outfile.addPage(p)
-            with open(fpath, 'wb') as f:
+            with open(fpath, "wb") as f:
                 outfile.write(f)
             layout, dim = get_page_layout(fpath)
             # fix rotated PDF
@@ -111,23 +119,25 @@ class PDFHandler(object):
             horizontal_text = get_text_objects(layout, ltype="horizontal_text")
             vertical_text = get_text_objects(layout, ltype="vertical_text")
             rotation = get_rotation(chars, horizontal_text, vertical_text)
-            if rotation != '':
-                fpath_new = ''.join([froot.replace('page', 'p'), '_rotated', fext])
+            if rotation != "":
+                fpath_new = "".join([froot.replace("page", "p"), "_rotated", fext])
                 os.rename(fpath, fpath_new)
-                infile = PdfFileReader(open(fpath_new, 'rb'), strict=False)
+                infile = PdfFileReader(open(fpath_new, "rb"), strict=False)
                 if infile.isEncrypted:
                     infile.decrypt(self.password)
                 outfile = PdfFileWriter()
                 p = infile.getPage(0)
-                if rotation == 'anticlockwise':
+                if rotation == "anticlockwise":
                     p.rotateClockwise(90)
-                elif rotation == 'clockwise':
+                elif rotation == "clockwise":
                     p.rotateCounterClockwise(90)
                 outfile.addPage(p)
-                with open(fpath, 'wb') as f:
+                with open(fpath, "wb") as f:
                     outfile.write(f)
 
-    def parse(self, flavor='lattice', suppress_stdout=False, layout_kwargs={}, **kwargs):
+    def parse(
+        self, flavor="lattice", suppress_stdout=False, layout_kwargs={}, **kwargs
+    ):
         """Extracts tables by calling parser.get_tables on all single
         page PDFs.
 
@@ -153,11 +163,13 @@ class PDFHandler(object):
         with TemporaryDirectory() as tempdir:
             for p in self.pages:
                 self._save_page(self.filepath, p, tempdir)
-            pages = [os.path.join(tempdir, 'page-{0}.pdf'.format(p))
-                     for p in self.pages]
-            parser = Lattice(**kwargs) if flavor == 'lattice' else Stream(**kwargs)
+            pages = [
+                os.path.join(tempdir, f"page-{p}.pdf") for p in self.pages
+            ]
+            parser = Lattice(**kwargs) if flavor == "lattice" else Stream(**kwargs)
             for p in pages:
-                t = parser.extract_tables(p, suppress_stdout=suppress_stdout,
-                                          layout_kwargs=layout_kwargs)
+                t = parser.extract_tables(
+                    p, suppress_stdout=suppress_stdout, layout_kwargs=layout_kwargs
+                )
                 tables.extend(t)
-        return TableList(tables)
+        return TableList(sorted(tables))

camelot/image_processing.py

@@ -1,7 +1,5 @@
 # -*- coding: utf-8 -*-
 
-from __future__ import division
-
 import cv2
 import numpy as np
 
@@ -39,17 +37,23 @@ def adaptive_threshold(imagename, process_background=False, blocksize=15, c=-2):
 
     if process_background:
         threshold = cv2.adaptiveThreshold(
-            gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
-            cv2.THRESH_BINARY, blocksize, c)
+            gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, blocksize, c
+        )
     else:
         threshold = cv2.adaptiveThreshold(
-            np.invert(gray), 255,
-            cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, blocksize, c)
+            np.invert(gray),
+            255,
+            cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
+            cv2.THRESH_BINARY,
+            blocksize,
+            c,
+        )
     return img, threshold
 
 
-def find_lines(threshold, regions=None, direction='horizontal',
-               line_scale=15, iterations=0):
+def find_lines(
+    threshold, regions=None, direction="horizontal", line_scale=15, iterations=0
+):
     """Finds horizontal and vertical lines by applying morphological
     transformations on an image.
 
@@ -87,15 +91,14 @@ def find_lines(threshold, regions=None, direction='horizontal',
     """
     lines = []
 
-    if direction == 'vertical':
+    if direction == "vertical":
         size = threshold.shape[0] // line_scale
         el = cv2.getStructuringElement(cv2.MORPH_RECT, (1, size))
-    elif direction == 'horizontal':
+    elif direction == "horizontal":
         size = threshold.shape[1] // line_scale
         el = cv2.getStructuringElement(cv2.MORPH_RECT, (size, 1))
     elif direction is None:
-        raise ValueError("Specify direction as either 'vertical' or"
-                         " 'horizontal'")
+        raise ValueError("Specify direction as either 'vertical' or 'horizontal'")
 
     if regions is not None:
         region_mask = np.zeros(threshold.shape)
@@ -110,19 +113,21 @@ def find_lines(threshold, regions=None, direction='horizontal',
 
     try:
         _, contours, _ = cv2.findContours(
-            threshold.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
+            threshold.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
+        )
     except ValueError:
         # for opencv backward compatibility
         contours, _ = cv2.findContours(
-            threshold.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
+            threshold.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
+        )
 
     for c in contours:
         x, y, w, h = cv2.boundingRect(c)
         x1, x2 = x, x + w
         y1, y2 = y, y + h
-        if direction == 'vertical':
+        if direction == "vertical":
             lines.append(((x1 + x2) // 2, y2, (x1 + x2) // 2, y1))
-        elif direction == 'horizontal':
+        elif direction == "horizontal":
             lines.append((x1, (y1 + y2) // 2, x2, (y1 + y2) // 2))
 
     return dmask, lines
@@ -150,11 +155,13 @@ def find_contours(vertical, horizontal):
 
     try:
         __, contours, __ = cv2.findContours(
-            mask.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
+            mask.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
+        )
     except ValueError:
         # for opencv backward compatibility
         contours, __ = cv2.findContours(
-            mask.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
+            mask.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
+        )
     # sort in reverse based on contour area and use first 10 contours
     contours = sorted(contours, key=cv2.contourArea, reverse=True)[:10]
 
@@ -196,11 +203,13 @@ def find_joints(contours, vertical, horizontal):
         roi = joints[y : y + h, x : x + w]
         try:
             __, jc, __ = cv2.findContours(
-                roi.astype(np.uint8), cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)
+                roi.astype(np.uint8), cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE
+            )
         except ValueError:
             # for opencv backward compatibility
             jc, __ = cv2.findContours(
-                roi.astype(np.uint8), cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE)
+                roi.astype(np.uint8), cv2.RETR_CCOMP, cv2.CHAIN_APPROX_SIMPLE
+            )
         if len(jc) <= 4:  # remove contours with less than 4 joints
             continue
         joint_coords = []

camelot/io.py

@@ -6,8 +6,15 @@ from .handlers import PDFHandler
 from .utils import validate_input, remove_extra
 
 
-def read_pdf(filepath, pages='1', password=None, flavor='lattice',
-             suppress_stdout=False, layout_kwargs={}, **kwargs):
+def read_pdf(
+    filepath,
+    pages="1",
+    password=None,
+    flavor="lattice",
+    suppress_stdout=False,
+    layout_kwargs={},
+    **kwargs
+):
     """Read PDF and return extracted tables.
 
     Note: kwargs annotated with ^ can only be used with flavor='stream'
@@ -91,9 +98,10 @@ def read_pdf(filepath, pages='1', password=None, flavor='lattice',
     tables : camelot.core.TableList
 
     """
-    if flavor not in ['lattice', 'stream']:
-        raise NotImplementedError("Unknown flavor specified."
-                                  " Use either 'lattice' or 'stream'")
+    if flavor not in ["lattice", "stream"]:
+        raise NotImplementedError(
+            "Unknown flavor specified." " Use either 'lattice' or 'stream'"
+        )
 
     with warnings.catch_warnings():
         if suppress_stdout:
@@ -102,6 +110,10 @@ def read_pdf(filepath, pages='1', password=None, flavor='lattice',
         validate_input(kwargs, flavor=flavor)
         p = PDFHandler(filepath, pages=pages, password=password)
         kwargs = remove_extra(kwargs, flavor=flavor)
-        tables = p.parse(flavor=flavor, suppress_stdout=suppress_stdout,
-                         layout_kwargs=layout_kwargs, **kwargs)
+        tables = p.parse(
+            flavor=flavor,
+            suppress_stdout=suppress_stdout,
+            layout_kwargs=layout_kwargs,
+            **kwargs
+        )
         return tables

camelot/parsers/base.py

@@ -8,13 +8,13 @@ from ..utils import get_page_layout, get_text_objects
 class BaseParser(object):
     """Defines a base parser.
     """
+
     def _generate_layout(self, filename, layout_kwargs):
         self.filename = filename
         self.layout_kwargs = layout_kwargs
-        self.layout, self.dimensions = get_page_layout(
-            filename, **layout_kwargs)
-        self.images = get_text_objects(self.layout, ltype='image')
-        self.horizontal_text = get_text_objects(self.layout, ltype='horizontal_text')
-        self.vertical_text = get_text_objects(self.layout, ltype='vertical_text')
+        self.layout, self.dimensions = get_page_layout(filename, **layout_kwargs)
+        self.images = get_text_objects(self.layout, ltype="image")
+        self.horizontal_text = get_text_objects(self.layout, ltype="horizontal_text")
+        self.vertical_text = get_text_objects(self.layout, ltype="vertical_text")
         self.pdf_width, self.pdf_height = self.dimensions
         self.rootname, __ = os.path.splitext(self.filename)

camelot/parsers/lattice.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 
-from __future__ import division
 import os
 import sys
 import copy
@@ -14,14 +13,25 @@ import pandas as pd
 
 from .base import BaseParser
 from ..core import Table
-from ..utils import (scale_image, scale_pdf, segments_in_bbox, text_in_bbox,
-                     merge_close_lines, get_table_index, compute_accuracy,
-                     compute_whitespace)
-from ..image_processing import (adaptive_threshold, find_lines,
-                                find_contours, find_joints)
+from ..utils import (
+    scale_image,
+    scale_pdf,
+    segments_in_bbox,
+    text_in_bbox,
+    merge_close_lines,
+    get_table_index,
+    compute_accuracy,
+    compute_whitespace,
+)
+from ..image_processing import (
+    adaptive_threshold,
+    find_lines,
+    find_contours,
+    find_joints,
+)
 
 
-logger = logging.getLogger('camelot')
+logger = logging.getLogger("camelot")
 
 
 class Lattice(BaseParser):
@@ -83,11 +93,26 @@ class Lattice(BaseParser):
         Resolution used for PDF to PNG conversion.
 
     """
-    def __init__(self, table_regions=None, table_areas=None, process_background=False,
-                 line_scale=15, copy_text=None, shift_text=['l', 't'],
-                 split_text=False, flag_size=False, strip_text='', line_tol=2,
-                 joint_tol=2, threshold_blocksize=15, threshold_constant=-2,
-                 iterations=0, resolution=300, **kwargs):
+
+    def __init__(
+        self,
+        table_regions=None,
+        table_areas=None,
+        process_background=False,
+        line_scale=15,
+        copy_text=None,
+        shift_text=["l", "t"],
+        split_text=False,
+        flag_size=False,
+        strip_text="",
+        line_tol=2,
+        joint_tol=2,
+        threshold_blocksize=15,
+        threshold_constant=-2,
+        iterations=0,
+        resolution=300,
+        **kwargs
+    ):
         self.table_regions = table_regions
         self.table_areas = table_areas
         self.process_background = process_background
@@ -130,19 +155,19 @@ class Lattice(BaseParser):
         indices = []
         for r_idx, c_idx, text in idx:
             for d in shift_text:
-                if d == 'l':
+                if d == "l":
                     if t.cells[r_idx][c_idx].hspan:
                         while not t.cells[r_idx][c_idx].left:
                             c_idx -= 1
-                if d == 'r':
+                if d == "r":
                     if t.cells[r_idx][c_idx].hspan:
                         while not t.cells[r_idx][c_idx].right:
                             c_idx += 1
-                if d == 't':
+                if d == "t":
                     if t.cells[r_idx][c_idx].vspan:
                         while not t.cells[r_idx][c_idx].top:
                             r_idx -= 1
-                if d == 'b':
+                if d == "b":
                     if t.cells[r_idx][c_idx].vspan:
                         while not t.cells[r_idx][c_idx].bottom:
                             r_idx += 1
@@ -171,13 +196,13 @@ class Lattice(BaseParser):
             if f == "h":
                 for i in range(len(t.cells)):
                     for j in range(len(t.cells[i])):
-                        if t.cells[i][j].text.strip() == '':
+                        if t.cells[i][j].text.strip() == "":
                             if t.cells[i][j].hspan and not t.cells[i][j].left:
                                 t.cells[i][j].text = t.cells[i][j - 1].text
             elif f == "v":
                 for i in range(len(t.cells)):
                     for j in range(len(t.cells[i])):
-                        if t.cells[i][j].text.strip() == '':
+                        if t.cells[i][j].text.strip() == "":
                             if t.cells[i][j].vspan and not t.cells[i][j].top:
                                 t.cells[i][j].text = t.cells[i - 1][j].text
         return t
@@ -185,11 +210,12 @@ class Lattice(BaseParser):
     def _generate_image(self):
         from ..ext.ghostscript import Ghostscript
 
-        self.imagename = ''.join([self.rootname, '.png'])
-        gs_call = '-q -sDEVICE=png16m -o {} -r300 {}'.format(
-            self.imagename, self.filename)
+        self.imagename = "".join([self.rootname, ".png"])
+        gs_call = "-q -sDEVICE=png16m -o {} -r300 {}".format(
+            self.imagename, self.filename
+        )
         gs_call = gs_call.encode().split()
-        null = open(os.devnull, 'wb')
+        null = open(os.devnull, "wb")
         with Ghostscript(*gs_call, stdout=null) as gs:
             pass
         null.close()
@@ -208,8 +234,11 @@ class Lattice(BaseParser):
             return scaled_areas
 
         self.image, self.threshold = adaptive_threshold(
-            self.imagename, process_background=self.process_background,
-            blocksize=self.threshold_blocksize, c=self.threshold_constant)
+            self.imagename,
+            process_background=self.process_background,
+            blocksize=self.threshold_blocksize,
+            c=self.threshold_constant,
+        )
 
         image_width = self.image.shape[1]
         image_height = self.image.shape[0]
@@ -226,21 +255,35 @@ class Lattice(BaseParser):
                 regions = scale_areas(self.table_regions)
 
             vertical_mask, vertical_segments = find_lines(
-                self.threshold, regions=regions, direction='vertical',
-                line_scale=self.line_scale, iterations=self.iterations)
+                self.threshold,
+                regions=regions,
+                direction="vertical",
+                line_scale=self.line_scale,
+                iterations=self.iterations,
+            )
             horizontal_mask, horizontal_segments = find_lines(
-                self.threshold, regions=regions, direction='horizontal',
-                line_scale=self.line_scale, iterations=self.iterations)
+                self.threshold,
+                regions=regions,
+                direction="horizontal",
+                line_scale=self.line_scale,
+                iterations=self.iterations,
+            )
 
             contours = find_contours(vertical_mask, horizontal_mask)
             table_bbox = find_joints(contours, vertical_mask, horizontal_mask)
         else:
             vertical_mask, vertical_segments = find_lines(
-                self.threshold, direction='vertical', line_scale=self.line_scale,
-                iterations=self.iterations)
+                self.threshold,
+                direction="vertical",
+                line_scale=self.line_scale,
+                iterations=self.iterations,
+            )
             horizontal_mask, horizontal_segments = find_lines(
-                self.threshold, direction='horizontal', line_scale=self.line_scale,
-                iterations=self.iterations)
+                self.threshold,
+                direction="horizontal",
+                line_scale=self.line_scale,
+                iterations=self.iterations,
+            )
 
             areas = scale_areas(self.table_areas)
             table_bbox = find_joints(areas, vertical_mask, horizontal_mask)
@@ -248,18 +291,20 @@ class Lattice(BaseParser):
         self.table_bbox_unscaled = copy.deepcopy(table_bbox)
 
         self.table_bbox, self.vertical_segments, self.horizontal_segments = scale_image(
-            table_bbox, vertical_segments, horizontal_segments, pdf_scalers)
+            table_bbox, vertical_segments, horizontal_segments, pdf_scalers
+        )
 
     def _generate_columns_and_rows(self, table_idx, tk):
         # select elements which lie within table_bbox
         t_bbox = {}
         v_s, h_s = segments_in_bbox(
-            tk, self.vertical_segments, self.horizontal_segments)
-        t_bbox['horizontal'] = text_in_bbox(tk, self.horizontal_text)
-        t_bbox['vertical'] = text_in_bbox(tk, self.vertical_text)
+            tk, self.vertical_segments, self.horizontal_segments
+        )
+        t_bbox["horizontal"] = text_in_bbox(tk, self.horizontal_text)
+        t_bbox["vertical"] = text_in_bbox(tk, self.vertical_text)
 
-        t_bbox['horizontal'].sort(key=lambda x: (-x.y0, x.x0))
-        t_bbox['vertical'].sort(key=lambda x: (x.x0, -x.y0))
+        t_bbox["horizontal"].sort(key=lambda x: (-x.y0, x.x0))
+        t_bbox["vertical"].sort(key=lambda x: (x.x0, -x.y0))
 
         self.t_bbox = t_bbox
 
@@ -268,23 +313,19 @@ class Lattice(BaseParser):
         cols.extend([tk[0], tk[2]])
         rows.extend([tk[1], tk[3]])
         # sort horizontal and vertical segments
-        cols = merge_close_lines(
-            sorted(cols), line_tol=self.line_tol)
-        rows = merge_close_lines(
-            sorted(rows, reverse=True), line_tol=self.line_tol)
+        cols = merge_close_lines(sorted(cols), line_tol=self.line_tol)
+        rows = merge_close_lines(sorted(rows, reverse=True), line_tol=self.line_tol)
         # make grid using x and y coord of shortlisted rows and cols
-        cols = [(cols[i], cols[i + 1])
-                for i in range(0, len(cols) - 1)]
-        rows = [(rows[i], rows[i + 1])
-                for i in range(0, len(rows) - 1)]
+        cols = [(cols[i], cols[i + 1]) for i in range(0, len(cols) - 1)]
+        rows = [(rows[i], rows[i + 1]) for i in range(0, len(rows) - 1)]
 
         return cols, rows, v_s, h_s
 
     def _generate_table(self, table_idx, cols, rows, **kwargs):
-        v_s = kwargs.get('v_s')
-        h_s = kwargs.get('h_s')
+        v_s = kwargs.get("v_s")
+        h_s = kwargs.get("h_s")
         if v_s is None or h_s is None:
-            raise ValueError('No segments found on {}'.format(self.rootname))
+            raise ValueError("No segments found on {}".format(self.rootname))
 
         table = Table(cols, rows)
         # set table edges to True using ver+hor lines
@@ -297,14 +338,21 @@ class Lattice(BaseParser):
         pos_errors = []
         # TODO: have a single list in place of two directional ones?
         # sorted on x-coordinate based on reading order i.e. LTR or RTL
-        for direction in ['vertical', 'horizontal']:
+        for direction in ["vertical", "horizontal"]:
             for t in self.t_bbox[direction]:
                 indices, error = get_table_index(
-                    table, t, direction, split_text=self.split_text,
-                    flag_size=self.flag_size, strip_text=self.strip_text)
+                    table,
+                    t,
+                    direction,
+                    split_text=self.split_text,
+                    flag_size=self.flag_size,
+                    strip_text=self.strip_text,
+                )
                 if indices[:2] != (-1, -1):
                     pos_errors.append(error)
-                    indices = Lattice._reduce_index(table, indices, shift_text=self.shift_text)
+                    indices = Lattice._reduce_index(
+                        table, indices, shift_text=self.shift_text
+                    )
                     for r_idx, c_idx, text in indices:
                         table.cells[r_idx][c_idx].text = text
         accuracy = compute_accuracy([[100, pos_errors]])
@@ -317,11 +365,11 @@ class Lattice(BaseParser):
         table.shape = table.df.shape
 
         whitespace = compute_whitespace(data)
-        table.flavor = 'lattice'
+        table.flavor = "lattice"
         table.accuracy = accuracy
         table.whitespace = whitespace
         table.order = table_idx + 1
-        table.page = int(os.path.basename(self.rootname).replace('page-', ''))
+        table.page = int(os.path.basename(self.rootname).replace("page-", ""))
 
         # for plotting
         _text = []
@@ -337,15 +385,18 @@ class Lattice(BaseParser):
     def extract_tables(self, filename, suppress_stdout=False, layout_kwargs={}):
         self._generate_layout(filename, layout_kwargs)
         if not suppress_stdout:
-            logger.info('Processing {}'.format(os.path.basename(self.rootname)))
+            logger.info("Processing {}".format(os.path.basename(self.rootname)))
 
         if not self.horizontal_text:
             if self.images:
-                warnings.warn('{} is image-based, camelot only works on'
-                              ' text-based pages.'.format(os.path.basename(self.rootname)))
+                warnings.warn(
+                    "{} is image-based, camelot only works on"
+                    " text-based pages.".format(os.path.basename(self.rootname))
+                )
             else:
-                warnings.warn('No tables found on {}'.format(
-                    os.path.basename(self.rootname)))
+                warnings.warn(
+                    "No tables found on {}".format(os.path.basename(self.rootname))
+                )
             return []
 
         self._generate_image()
@@ -353,8 +404,9 @@ class Lattice(BaseParser):
 
         _tables = []
         # sort tables based on y-coord
-        for table_idx, tk in enumerate(sorted(
-                self.table_bbox.keys(), key=lambda x: x[1], reverse=True)):
+        for table_idx, tk in enumerate(
+            sorted(self.table_bbox.keys(), key=lambda x: x[1], reverse=True)
+        ):
             cols, rows, v_s, h_s = self._generate_columns_and_rows(table_idx, tk)
             table = self._generate_table(table_idx, cols, rows, v_s=v_s, h_s=h_s)
             table._bbox = tk

camelot/parsers/stream.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 
-from __future__ import division
 import os
 import logging
 import warnings
@@ -10,11 +9,10 @@ import pandas as pd
 
 from .base import BaseParser
 from ..core import TextEdges, Table
-from ..utils import (text_in_bbox, get_table_index, compute_accuracy,
-                     compute_whitespace)
+from ..utils import text_in_bbox, get_table_index, compute_accuracy, compute_whitespace
 
 
-logger = logging.getLogger('camelot')
+logger = logging.getLogger("camelot")
 
 
 class Stream(BaseParser):
@@ -55,9 +53,20 @@ class Stream(BaseParser):
         to generate columns.
 
     """
-    def __init__(self, table_regions=None, table_areas=None, columns=None, split_text=False,
-                 flag_size=False, strip_text='', edge_tol=50, row_tol=2,
-                 column_tol=0, **kwargs):
+
+    def __init__(
+        self,
+        table_regions=None,
+        table_areas=None,
+        columns=None,
+        split_text=False,
+        flag_size=False,
+        strip_text="",
+        edge_tol=50,
+        row_tol=2,
+        column_tol=0,
+        **kwargs
+    ):
         self.table_regions = table_regions
         self.table_areas = table_areas
         self.columns = columns
@@ -150,8 +159,9 @@ class Stream(BaseParser):
             else:
                 lower = merged[-1]
                 if column_tol >= 0:
-                    if (higher[0] <= lower[1] or
-                            np.isclose(higher[0], lower[1], atol=column_tol)):
+                    if higher[0] <= lower[1] or np.isclose(
+                        higher[0], lower[1], atol=column_tol
+                    ):
                         upper_bound = max(lower[1], higher[1])
                         lower_bound = min(lower[0], higher[0])
                         merged[-1] = (lower_bound, upper_bound)
@@ -186,13 +196,14 @@ class Stream(BaseParser):
             List of continuous row y-coordinate tuples.
 
         """
-        row_mids = [sum([(t.y0 + t.y1) / 2 for t in r]) / len(r)
-                    if len(r) > 0 else 0 for r in rows_grouped]
+        row_mids = [
+            sum([(t.y0 + t.y1) / 2 for t in r]) / len(r) if len(r) > 0 else 0
+            for r in rows_grouped
+        ]
         rows = [(row_mids[i] + row_mids[i - 1]) / 2 for i in range(1, len(row_mids))]
         rows.insert(0, text_y_max)
         rows.append(text_y_min)
-        rows = [(rows[i], rows[i + 1])
-                for i in range(0, len(rows) - 1)]
+        rows = [(rows[i], rows[i + 1]) for i in range(0, len(rows) - 1)]
         return rows
 
     @staticmethod
@@ -217,8 +228,9 @@ class Stream(BaseParser):
         if text:
             text = Stream._group_rows(text, row_tol=row_tol)
             elements = [len(r) for r in text]
-            new_cols = [(t.x0, t.x1)
-                        for r in text if len(r) == max(elements) for t in r]
+            new_cols = [
+                (t.x0, t.x1) for r in text if len(r) == max(elements) for t in r
+            ]
             cols.extend(Stream._merge_columns(sorted(new_cols)))
         return cols
 
@@ -243,15 +255,13 @@ class Stream(BaseParser):
         cols = [(cols[i][0] + cols[i - 1][1]) / 2 for i in range(1, len(cols))]
         cols.insert(0, text_x_min)
         cols.append(text_x_max)
-        cols = [(cols[i], cols[i + 1])
-                for i in range(0, len(cols) - 1)]
+        cols = [(cols[i], cols[i + 1]) for i in range(0, len(cols) - 1)]
         return cols
 
     def _validate_columns(self):
         if self.table_areas is not None and self.columns is not None:
             if len(self.table_areas) != len(self.columns):
-                raise ValueError("Length of table_areas and columns"
-                                 " should be equal")
+                raise ValueError("Length of table_areas and columns" " should be equal")
 
     def _nurminen_table_detection(self, textlines):
         """A general implementation of the table detection algorithm
@@ -286,7 +296,11 @@ class Stream(BaseParser):
                 # filter horizontal text
                 hor_text = []
                 for region in self.table_regions:
-                    x1, y1, x2, y2 = region
+                    x1, y1, x2, y2 = region.split(",")
+                    x1 = float(x1)
+                    y1 = float(y1)
+                    x2 = float(x2)
+                    y2 = float(y2)
                     region_text = text_in_bbox((x1, y2, x2, y1), self.horizontal_text)
                     hor_text.extend(region_text)
             # find tables based on nurminen's detection algorithm
@@ -305,16 +319,16 @@ class Stream(BaseParser):
     def _generate_columns_and_rows(self, table_idx, tk):
         # select elements which lie within table_bbox
         t_bbox = {}
-        t_bbox['horizontal'] = text_in_bbox(tk, self.horizontal_text)
-        t_bbox['vertical'] = text_in_bbox(tk, self.vertical_text)
+        t_bbox["horizontal"] = text_in_bbox(tk, self.horizontal_text)
+        t_bbox["vertical"] = text_in_bbox(tk, self.vertical_text)
 
-        t_bbox['horizontal'].sort(key=lambda x: (-x.y0, x.x0))
-        t_bbox['vertical'].sort(key=lambda x: (x.x0, -x.y0))
+        t_bbox["horizontal"].sort(key=lambda x: (-x.y0, x.x0))
+        t_bbox["vertical"].sort(key=lambda x: (x.x0, -x.y0))
 
         self.t_bbox = t_bbox
 
         text_x_min, text_y_min, text_x_max, text_y_max = self._text_bbox(self.t_bbox)
-        rows_grouped = self._group_rows(self.t_bbox['horizontal'], row_tol=self.row_tol)
+        rows_grouped = self._group_rows(self.t_bbox["horizontal"], row_tol=self.row_tol)
         rows = self._join_rows(rows_grouped, text_y_max, text_y_min)
         elements = [len(r) for r in rows_grouped]
 
@@ -323,7 +337,7 @@ class Stream(BaseParser):
             # take (0, pdf_width) by default
             # similar to else condition
             # len can't be 1
-            cols = self.columns[table_idx].split(',')
+            cols = self.columns[table_idx].split(",")
             cols = [float(c) for c in cols]
             cols.insert(0, text_x_min)
             cols.append(text_x_max)
@@ -342,20 +356,29 @@ class Stream(BaseParser):
                 if len(elements):
                     ncols = max(set(elements), key=elements.count)
                 else:
-                    warnings.warn("No tables found in table area {}".format(
-                        table_idx + 1))
+                    warnings.warn(
+                        f"No tables found in table area {table_idx + 1}"
+                    )
             cols = [(t.x0, t.x1) for r in rows_grouped if len(r) == ncols for t in r]
             cols = self._merge_columns(sorted(cols), column_tol=self.column_tol)
             inner_text = []
             for i in range(1, len(cols)):
                 left = cols[i - 1][1]
                 right = cols[i][0]
-                inner_text.extend([t for direction in self.t_bbox
-                                     for t in self.t_bbox[direction]
-                                     if t.x0 > left and t.x1 < right])
-            outer_text = [t for direction in self.t_bbox
-                            for t in self.t_bbox[direction]
-                            if t.x0 > cols[-1][1] or t.x1 < cols[0][0]]
+                inner_text.extend(
+                    [
+                        t
+                        for direction in self.t_bbox
+                        for t in self.t_bbox[direction]
+                        if t.x0 > left and t.x1 < right
+                    ]
+                )
+            outer_text = [
+                t
+                for direction in self.t_bbox
+                for t in self.t_bbox[direction]
+                if t.x0 > cols[-1][1] or t.x1 < cols[0][0]
+            ]
             inner_text.extend(outer_text)
             cols = self._add_columns(cols, inner_text, self.row_tol)
             cols = self._join_columns(cols, text_x_min, text_x_max)
@@ -369,11 +392,16 @@ class Stream(BaseParser):
         pos_errors = []
         # TODO: have a single list in place of two directional ones?
         # sorted on x-coordinate based on reading order i.e. LTR or RTL
-        for direction in ['vertical', 'horizontal']:
+        for direction in ["vertical", "horizontal"]:
             for t in self.t_bbox[direction]:
                 indices, error = get_table_index(
-                    table, t, direction, split_text=self.split_text,
-                    flag_size=self.flag_size, strip_text=self.strip_text)
+                    table,
+                    t,
+                    direction,
+                    split_text=self.split_text,
+                    flag_size=self.flag_size,
+                    strip_text=self.strip_text,
+                )
                 if indices[:2] != (-1, -1):
                     pos_errors.append(error)
                     for r_idx, c_idx, text in indices:
@@ -385,11 +413,11 @@ class Stream(BaseParser):
         table.shape = table.df.shape
 
         whitespace = compute_whitespace(data)
-        table.flavor = 'stream'
+        table.flavor = "stream"
         table.accuracy = accuracy
         table.whitespace = whitespace
         table.order = table_idx + 1
-        table.page = int(os.path.basename(self.rootname).replace('page-', ''))
+        table.page = int(os.path.basename(self.rootname).replace("page-", ""))
 
         # for plotting
         _text = []
@@ -404,24 +432,28 @@ class Stream(BaseParser):
 
     def extract_tables(self, filename, suppress_stdout=False, layout_kwargs={}):
         self._generate_layout(filename, layout_kwargs)
+        base_filename = os.path.basename(self.rootname)
+
         if not suppress_stdout:
-            logger.info('Processing {}'.format(os.path.basename(self.rootname)))
+            logger.info(f"Processing {base_filename}")
 
         if not self.horizontal_text:
             if self.images:
-                warnings.warn('{} is image-based, camelot only works on'
-                              ' text-based pages.'.format(os.path.basename(self.rootname)))
+                warnings.warn(
+                    f"{base_filename} is image-based, camelot only works on"
+                    " text-based pages."
+                )
             else:
-                warnings.warn('No tables found on {}'.format(
-                    os.path.basename(self.rootname)))
+                warnings.warn(f"No tables found on {base_filename}")
             return []
 
         self._generate_table_bbox()
 
         _tables = []
         # sort tables based on y-coord
-        for table_idx, tk in enumerate(sorted(
-                self.table_bbox.keys(), key=lambda x: x[1], reverse=True)):
+        for table_idx, tk in enumerate(
+            sorted(self.table_bbox.keys(), key=lambda x: x[1], reverse=True)
+        ):
             cols, rows = self._generate_columns_and_rows(table_idx, tk)
             table = self._generate_table(table_idx, cols, rows)
             table._bbox = tk

camelot/plotting.py

@@ -10,7 +10,7 @@ else:
 
 
 class PlotMethods(object):
-    def __call__(self, table, kind='text', filename=None):
+    def __call__(self, table, kind="text", filename=None):
         """Plot elements found on PDF page based on kind
         specified, useful for debugging and playing with different
         parameters to get the best output.
@@ -31,14 +31,16 @@ class PlotMethods(object):
 
         """
         if not _HAS_MPL:
-            raise ImportError('matplotlib is required for plotting.')
+            raise ImportError("matplotlib is required for plotting.")
 
-        if table.flavor == 'lattice' and kind in ['textedge']:
-            raise NotImplementedError("Lattice flavor does not support kind='{}'".format(
-                                      kind))
-        elif table.flavor == 'stream' and kind in ['joint', 'line']:
-            raise NotImplementedError("Stream flavor does not support kind='{}'".format(
-                                      kind))
+        if table.flavor == "lattice" and kind in ["textedge"]:
+            raise NotImplementedError(
+                f"Lattice flavor does not support kind='{kind}'"
+            )
+        elif table.flavor == "stream" and kind in ["joint", "line"]:
+            raise NotImplementedError(
+                f"Stream flavor does not support kind='{kind}'"
+            )
 
         plot_method = getattr(self, kind)
         return plot_method(table)
@@ -57,18 +59,12 @@ class PlotMethods(object):
 
         """
         fig = plt.figure()
-        ax = fig.add_subplot(111, aspect='equal')
+        ax = fig.add_subplot(111, aspect="equal")
         xs, ys = [], []
         for t in table._text:
             xs.extend([t[0], t[2]])
             ys.extend([t[1], t[3]])
-            ax.add_patch(
-                patches.Rectangle(
-                    (t[0], t[1]),
-                    t[2] - t[0],
-                    t[3] - t[1]
-                )
-            )
+            ax.add_patch(patches.Rectangle((t[0], t[1]), t[2] - t[0], t[3] - t[1]))
         ax.set_xlim(min(xs) - 10, max(xs) + 10)
         ax.set_ylim(min(ys) - 10, max(ys) + 10)
         return fig
@@ -87,21 +83,17 @@ class PlotMethods(object):
 
         """
         fig = plt.figure()
-        ax = fig.add_subplot(111, aspect='equal')
+        ax = fig.add_subplot(111, aspect="equal")
         for row in table.cells:
             for cell in row:
                 if cell.left:
-                    ax.plot([cell.lb[0], cell.lt[0]],
-                            [cell.lb[1], cell.lt[1]])
+                    ax.plot([cell.lb[0], cell.lt[0]], [cell.lb[1], cell.lt[1]])
                 if cell.right:
-                    ax.plot([cell.rb[0], cell.rt[0]],
-                            [cell.rb[1], cell.rt[1]])
+                    ax.plot([cell.rb[0], cell.rt[0]], [cell.rb[1], cell.rt[1]])
                 if cell.top:
-                    ax.plot([cell.lt[0], cell.rt[0]],
-                            [cell.lt[1], cell.rt[1]])
+                    ax.plot([cell.lt[0], cell.rt[0]], [cell.lt[1], cell.rt[1]])
                 if cell.bottom:
-                    ax.plot([cell.lb[0], cell.rb[0]],
-                            [cell.lb[1], cell.rb[1]])
+                    ax.plot([cell.lb[0], cell.rb[0]], [cell.lb[1], cell.rb[1]])
         return fig
 
     def contour(self, table):
@@ -124,7 +116,7 @@ class PlotMethods(object):
             img, table_bbox = (None, {table._bbox: None})
             _FOR_LATTICE = False
         fig = plt.figure()
-        ax = fig.add_subplot(111, aspect='equal')
+        ax = fig.add_subplot(111, aspect="equal")
 
         xs, ys = [], []
         if not _FOR_LATTICE:
@@ -133,21 +125,14 @@ class PlotMethods(object):
                 ys.extend([t[1], t[3]])
                 ax.add_patch(
                     patches.Rectangle(
-                        (t[0], t[1]),
-                        t[2] - t[0],
-                        t[3] - t[1],
-                        color='blue'
+                        (t[0], t[1]), t[2] - t[0], t[3] - t[1], color="blue"
                     )
                 )
 
         for t in table_bbox.keys():
             ax.add_patch(
                 patches.Rectangle(
-                    (t[0], t[1]),
-                    t[2] - t[0],
-                    t[3] - t[1],
-                    fill=False,
-                    color='red'
+                    (t[0], t[1]), t[2] - t[0], t[3] - t[1], fill=False, color="red"
                 )
             )
             if not _FOR_LATTICE:
@@ -173,25 +158,19 @@ class PlotMethods(object):
 
         """
         fig = plt.figure()
-        ax = fig.add_subplot(111, aspect='equal')
+        ax = fig.add_subplot(111, aspect="equal")
         xs, ys = [], []
         for t in table._text:
             xs.extend([t[0], t[2]])
             ys.extend([t[1], t[3]])
             ax.add_patch(
-                patches.Rectangle(
-                    (t[0], t[1]),
-                    t[2] - t[0],
-                    t[3] - t[1],
-                    color='blue'
-                )
+                patches.Rectangle((t[0], t[1]), t[2] - t[0], t[3] - t[1], color="blue")
             )
         ax.set_xlim(min(xs) - 10, max(xs) + 10)
         ax.set_ylim(min(ys) - 10, max(ys) + 10)
 
         for te in table._textedges:
-            ax.plot([te.x, te.x],
-                    [te.y0, te.y1])
+            ax.plot([te.x, te.x], [te.y0, te.y1])
 
         return fig
 
@@ -210,14 +189,14 @@ class PlotMethods(object):
         """
         img, table_bbox = table._image
         fig = plt.figure()
-        ax = fig.add_subplot(111, aspect='equal')
+        ax = fig.add_subplot(111, aspect="equal")
         x_coord = []
         y_coord = []
         for k in table_bbox.keys():
             for coord in table_bbox[k]:
                 x_coord.append(coord[0])
                 y_coord.append(coord[1])
-        ax.plot(x_coord, y_coord, 'ro')
+        ax.plot(x_coord, y_coord, "ro")
         ax.imshow(img)
         return fig
 
@@ -235,7 +214,7 @@ class PlotMethods(object):
 
         """
         fig = plt.figure()
-        ax = fig.add_subplot(111, aspect='equal')
+        ax = fig.add_subplot(111, aspect="equal")
         vertical, horizontal = table._segments
         for v in vertical:
             ax.plot([v[0], v[2]], [v[1], v[3]])

camelot/utils.py

@@ -1,8 +1,7 @@
 # -*- coding: utf-8 -*-
-from __future__ import division
 
 import os
-import sys
+import re
 import random
 import shutil
 import string
@@ -19,23 +18,22 @@ from pdfminer.pdfpage import PDFTextExtractionNotAllowed
 from pdfminer.pdfinterp import PDFResourceManager
 from pdfminer.pdfinterp import PDFPageInterpreter
 from pdfminer.converter import PDFPageAggregator
-from pdfminer.layout import (LAParams, LTAnno, LTChar, LTTextLineHorizontal,
-                             LTTextLineVertical, LTImage)
+from pdfminer.layout import (
+    LAParams,
+    LTAnno,
+    LTChar,
+    LTTextLineHorizontal,
+    LTTextLineVertical,
+    LTImage,
+)
 
-
-PY3 = sys.version_info[0] >= 3
-if PY3:
-    from urllib.request import urlopen
-    from urllib.parse import urlparse as parse_url
-    from urllib.parse import uses_relative, uses_netloc, uses_params
-else:
-    from urllib2 import urlopen
-    from urlparse import urlparse as parse_url
-    from urlparse import uses_relative, uses_netloc, uses_params
+from urllib.request import Request, urlopen
+from urllib.parse import urlparse as parse_url
+from urllib.parse import uses_relative, uses_netloc, uses_params
 
 
 _VALID_URLS = set(uses_relative + uses_netloc + uses_params)
-_VALID_URLS.discard('')
+_VALID_URLS.discard("")
 
 
 # https://github.com/pandas-dev/pandas/blob/master/pandas/io/common.py
@@ -59,9 +57,11 @@ def is_url(url):
 
 
 def random_string(length):
-    ret = ''
+    ret = ""
     while length:
-        ret += random.choice(string.digits + string.ascii_lowercase + string.ascii_uppercase)
+        ret += random.choice(
+            string.digits + string.ascii_lowercase + string.ascii_uppercase
+        )
         length -= 1
     return ret
 
@@ -79,14 +79,13 @@ def download_url(url):
         Temporary filepath.
 
     """
-    filename = '{}.pdf'.format(random_string(6))
-    with tempfile.NamedTemporaryFile('wb', delete=False) as f:
-        obj = urlopen(url)
-        if PY3:
-            content_type = obj.info().get_content_type()
-        else:
-            content_type = obj.info().getheader('Content-Type')
-        if content_type != 'application/pdf':
+    filename = f"{random_string(6)}.pdf"
+    with tempfile.NamedTemporaryFile("wb", delete=False) as f:
+        headers = {"User-Agent": "Mozilla/5.0"}
+        request = Request(url, None, headers)
+        obj = urlopen(request)
+        content_type = obj.info().get_content_type()
+        if content_type != "application/pdf":
             raise NotImplementedError("File format not supported")
         f.write(obj.read())
     filepath = os.path.join(os.path.dirname(f.name), filename)
@@ -94,39 +93,37 @@ def download_url(url):
     return filepath
 
 
-stream_kwargs = [
-    'columns',
-    'row_tol',
-    'column_tol'
-]
+stream_kwargs = ["columns", "edge_tol", "row_tol", "column_tol"]
 lattice_kwargs = [
-    'process_background',
-    'line_scale',
-    'copy_text',
-    'shift_text',
-    'line_tol',
-    'joint_tol',
-    'threshold_blocksize',
-    'threshold_constant',
-    'iterations'
+    "process_background",
+    "line_scale",
+    "copy_text",
+    "shift_text",
+    "line_tol",
+    "joint_tol",
+    "threshold_blocksize",
+    "threshold_constant",
+    "iterations",
+    "resolution",
 ]
 
 
-def validate_input(kwargs, flavor='lattice'):
+def validate_input(kwargs, flavor="lattice"):
     def check_intersection(parser_kwargs, input_kwargs):
         isec = set(parser_kwargs).intersection(set(input_kwargs.keys()))
         if isec:
-            raise ValueError("{} cannot be used with flavor='{}'".format(
-                             ",".join(sorted(isec)), flavor))
+            raise ValueError(
+                f"{','.join(sorted(isec))} cannot be used with flavor='{flavor}'"
+            )
 
-    if flavor == 'lattice':
+    if flavor == "lattice":
         check_intersection(stream_kwargs, kwargs)
     else:
         check_intersection(lattice_kwargs, kwargs)
 
 
-def remove_extra(kwargs, flavor='lattice'):
-    if flavor == 'lattice':
+def remove_extra(kwargs, flavor="lattice"):
+    if flavor == "lattice":
         for key in kwargs.keys():
             if key in stream_kwargs:
                 kwargs.pop(key)
@@ -256,15 +253,19 @@ def scale_image(tables, v_segments, h_segments, factors):
     v_segments_new = []
     for v in v_segments:
         x1, x2 = scale(v[0], scaling_factor_x), scale(v[2], scaling_factor_x)
-        y1, y2 = scale(abs(translate(-img_y, v[1])), scaling_factor_y), scale(
-            abs(translate(-img_y, v[3])), scaling_factor_y)
+        y1, y2 = (
+            scale(abs(translate(-img_y, v[1])), scaling_factor_y),
+            scale(abs(translate(-img_y, v[3])), scaling_factor_y),
+        )
         v_segments_new.append((x1, y1, x2, y2))
 
     h_segments_new = []
     for h in h_segments:
         x1, x2 = scale(h[0], scaling_factor_x), scale(h[2], scaling_factor_x)
-        y1, y2 = scale(abs(translate(-img_y, h[1])), scaling_factor_y), scale(
-            abs(translate(-img_y, h[3])), scaling_factor_y)
+        y1, y2 = (
+            scale(abs(translate(-img_y, h[1])), scaling_factor_y),
+            scale(abs(translate(-img_y, h[3])), scaling_factor_y),
+        )
         h_segments_new.append((x1, y1, x2, y2))
 
     return tables_new, v_segments_new, h_segments_new
@@ -291,13 +292,13 @@ def get_rotation(chars, horizontal_text, vertical_text):
         rotated 90 degree clockwise.
 
     """
-    rotation = ''
+    rotation = ""
     hlen = len([t for t in horizontal_text if t.get_text().strip()])
     vlen = len([t for t in vertical_text if t.get_text().strip()])
     if hlen < vlen:
         clockwise = sum(t.matrix[1] < 0 and t.matrix[2] > 0 for t in chars)
         anticlockwise = sum(t.matrix[1] > 0 and t.matrix[2] < 0 for t in chars)
-        rotation = 'anticlockwise' if clockwise < anticlockwise else 'clockwise'
+        rotation = "anticlockwise" if clockwise < anticlockwise else "clockwise"
     return rotation
 
 
@@ -325,10 +326,16 @@ def segments_in_bbox(bbox, v_segments, h_segments):
     """
     lb = (bbox[0], bbox[1])
     rt = (bbox[2], bbox[3])
-    v_s = [v for v in v_segments if v[1] > lb[1] - 2 and
-           v[3] < rt[1] + 2 and lb[0] - 2 <= v[0] <= rt[0] + 2]
-    h_s = [h for h in h_segments if h[0] > lb[0] - 2 and
-           h[2] < rt[0] + 2 and lb[1] - 2 <= h[1] <= rt[1] + 2]
+    v_s = [
+        v
+        for v in v_segments
+        if v[1] > lb[1] - 2 and v[3] < rt[1] + 2 and lb[0] - 2 <= v[0] <= rt[0] + 2
+    ]
+    h_s = [
+        h
+        for h in h_segments
+        if h[0] > lb[0] - 2 and h[2] < rt[0] + 2 and lb[1] - 2 <= h[1] <= rt[1] + 2
+    ]
     return v_s, h_s
 
 
@@ -351,9 +358,12 @@ def text_in_bbox(bbox, text):
     """
     lb = (bbox[0], bbox[1])
     rt = (bbox[2], bbox[3])
-    t_bbox = [t for t in text if lb[0] - 2 <= (t.x0 + t.x1) / 2.0
-              <= rt[0] + 2 and lb[1] - 2 <= (t.y0 + t.y1) / 2.0
-              <= rt[1] + 2]
+    t_bbox = [
+        t
+        for t in text
+        if lb[0] - 2 <= (t.x0 + t.x1) / 2.0 <= rt[0] + 2
+        and lb[1] - 2 <= (t.y0 + t.y1) / 2.0 <= rt[1] + 2
+    ]
     return t_bbox
 
 
@@ -385,12 +395,33 @@ def merge_close_lines(ar, line_tol=2):
     return ret
 
 
+def text_strip(text, strip=""):
+    """Strips any characters in `strip` that are present in `text`.
+    Parameters
+    ----------
+    text : str
+        Text to process and strip.
+    strip : str, optional (default: '')
+        Characters that should be stripped from `text`.
+    Returns
+    -------
+    stripped : str
+    """
+    if not strip:
+        return text
+
+    stripped = re.sub(
+        fr"[{''.join(map(re.escape, strip))}]", "", text, re.UNICODE
+    )
+    return stripped
+
+
 # TODO: combine the following functions into a TextProcessor class which
 # applies corresponding transformations sequentially
 # (inspired from sklearn.pipeline.Pipeline)
 
 
-def flag_font_size(textline, direction, strip_text=''):
+def flag_font_size(textline, direction, strip_text=""):
     """Flags super/subscripts in text by enclosing them with <s></s>.
     May give false positives.
 
@@ -409,10 +440,18 @@ def flag_font_size(textline, direction, strip_text=''):
     fstring : string
 
     """
-    if direction == 'horizontal':
-        d = [(t.get_text(), np.round(t.height, decimals=6)) for t in textline if not isinstance(t, LTAnno)]
-    elif direction == 'vertical':
-        d = [(t.get_text(), np.round(t.width, decimals=6)) for t in textline if not isinstance(t, LTAnno)]
+    if direction == "horizontal":
+        d = [
+            (t.get_text(), np.round(t.height, decimals=6))
+            for t in textline
+            if not isinstance(t, LTAnno)
+        ]
+    elif direction == "vertical":
+        d = [
+            (t.get_text(), np.round(t.width, decimals=6))
+            for t in textline
+            if not isinstance(t, LTAnno)
+        ]
     l = [np.round(size, decimals=6) for text, size in d]
     if len(set(l)) > 1:
         flist = []
@@ -420,21 +459,21 @@ def flag_font_size(textline, direction, strip_text=''):
         for key, chars in groupby(d, itemgetter(1)):
             if key == min_size:
                 fchars = [t[0] for t in chars]
-                if ''.join(fchars).strip():
-                    fchars.insert(0, '<s>')
-                    fchars.append('</s>')
-                    flist.append(''.join(fchars))
+                if "".join(fchars).strip():
+                    fchars.insert(0, "<s>")
+                    fchars.append("</s>")
+                    flist.append("".join(fchars))
             else:
                 fchars = [t[0] for t in chars]
-                if ''.join(fchars).strip():
-                    flist.append(''.join(fchars))
-        fstring = ''.join(flist).strip(strip_text)
+                if "".join(fchars).strip():
+                    flist.append("".join(fchars))
+        fstring = "".join(flist)
     else:
-        fstring = ''.join([t.get_text() for t in textline]).strip(strip_text)
-    return fstring
+        fstring = "".join([t.get_text() for t in textline])
+    return text_strip(fstring, strip_text)
 
 
-def split_textline(table, textline, direction, flag_size=False, strip_text=''):
+def split_textline(table, textline, direction, flag_size=False, strip_text=""):
     """Splits PDFMiner LTTextLine into substrings if it spans across
     multiple rows/columns.
 
@@ -464,38 +503,70 @@ def split_textline(table, textline, direction, flag_size=False, strip_text=''):
     cut_text = []
     bbox = textline.bbox
     try:
-        if direction == 'horizontal' and not textline.is_empty():
-            x_overlap = [i for i, x in enumerate(table.cols) if x[0] <= bbox[2] and bbox[0] <= x[1]]
-            r_idx = [j for j, r in enumerate(table.rows) if r[1] <= (bbox[1] + bbox[3]) / 2 <= r[0]]
+        if direction == "horizontal" and not textline.is_empty():
+            x_overlap = [
+                i
+                for i, x in enumerate(table.cols)
+                if x[0] <= bbox[2] and bbox[0] <= x[1]
+            ]
+            r_idx = [
+                j
+                for j, r in enumerate(table.rows)
+                if r[1] <= (bbox[1] + bbox[3]) / 2 <= r[0]
+            ]
             r = r_idx[0]
-            x_cuts = [(c, table.cells[r][c].x2) for c in x_overlap if table.cells[r][c].right]
+            x_cuts = [
+                (c, table.cells[r][c].x2) for c in x_overlap if table.cells[r][c].right
+            ]
             if not x_cuts:
                 x_cuts = [(x_overlap[0], table.cells[r][-1].x2)]
             for obj in textline._objs:
                 row = table.rows[r]
                 for cut in x_cuts:
                     if isinstance(obj, LTChar):
-                        if (row[1] <= (obj.y0 + obj.y1) / 2 <= row[0] and
-                                (obj.x0 + obj.x1) / 2 <= cut[1]):
+                        if (
+                            row[1] <= (obj.y0 + obj.y1) / 2 <= row[0]
+                            and (obj.x0 + obj.x1) / 2 <= cut[1]
+                        ):
                             cut_text.append((r, cut[0], obj))
                             break
+                        else:
+                            # TODO: add test
+                            if cut == x_cuts[-1]:
+                                cut_text.append((r, cut[0] + 1, obj))
                     elif isinstance(obj, LTAnno):
                         cut_text.append((r, cut[0], obj))
-        elif direction == 'vertical' and not textline.is_empty():
-            y_overlap = [j for j, y in enumerate(table.rows) if y[1] <= bbox[3] and bbox[1] <= y[0]]
-            c_idx = [i for i, c in enumerate(table.cols) if c[0] <= (bbox[0] + bbox[2]) / 2 <= c[1]]
+        elif direction == "vertical" and not textline.is_empty():
+            y_overlap = [
+                j
+                for j, y in enumerate(table.rows)
+                if y[1] <= bbox[3] and bbox[1] <= y[0]
+            ]
+            c_idx = [
+                i
+                for i, c in enumerate(table.cols)
+                if c[0] <= (bbox[0] + bbox[2]) / 2 <= c[1]
+            ]
             c = c_idx[0]
-            y_cuts = [(r, table.cells[r][c].y1) for r in y_overlap if table.cells[r][c].bottom]
+            y_cuts = [
+                (r, table.cells[r][c].y1) for r in y_overlap if table.cells[r][c].bottom
+            ]
             if not y_cuts:
                 y_cuts = [(y_overlap[0], table.cells[-1][c].y1)]
             for obj in textline._objs:
                 col = table.cols[c]
                 for cut in y_cuts:
                     if isinstance(obj, LTChar):
-                        if (col[0] <= (obj.x0 + obj.x1) / 2 <= col[1] and
-                                (obj.y0 + obj.y1) / 2 >= cut[1]):
+                        if (
+                            col[0] <= (obj.x0 + obj.x1) / 2 <= col[1]
+                            and (obj.y0 + obj.y1) / 2 >= cut[1]
+                        ):
                             cut_text.append((cut[0], c, obj))
                             break
+                        else:
+                            # TODO: add test
+                            if cut == y_cuts[-1]:
+                                cut_text.append((cut[0] - 1, c, obj))
                     elif isinstance(obj, LTAnno):
                         cut_text.append((cut[0], c, obj))
     except IndexError:
@@ -503,15 +574,26 @@ def split_textline(table, textline, direction, flag_size=False, strip_text=''):
     grouped_chars = []
     for key, chars in groupby(cut_text, itemgetter(0, 1)):
         if flag_size:
-            grouped_chars.append((key[0], key[1],
-                flag_font_size([t[2] for t in chars], direction, strip_text=strip_text)))
+            grouped_chars.append(
+                (
+                    key[0],
+                    key[1],
+                    flag_font_size(
+                        [t[2] for t in chars], direction, strip_text=strip_text
+                    ),
+                )
+            )
         else:
             gchars = [t[2].get_text() for t in chars]
-            grouped_chars.append((key[0], key[1], ''.join(gchars).strip(strip_text)))
+            grouped_chars.append(
+                (key[0], key[1], text_strip("".join(gchars), strip_text))
+            )
     return grouped_chars
 
 
-def get_table_index(table, t, direction, split_text=False, flag_size=False, strip_text='',):
+def get_table_index(
+    table, t, direction, split_text=False, flag_size=False, strip_text=""
+):
     """Gets indices of the table cell where given text object lies by
     comparing their y and x-coordinates.
 
@@ -550,8 +632,9 @@ def get_table_index(table, t, direction, split_text=False, flag_size=False, stri
     """
     r_idx, c_idx = [-1] * 2
     for r in range(len(table.rows)):
-        if ((t.y0 + t.y1) / 2.0 < table.rows[r][0] and
-                (t.y0 + t.y1) / 2.0 > table.rows[r][1]):
+        if (t.y0 + t.y1) / 2.0 < table.rows[r][0] and (t.y0 + t.y1) / 2.0 > table.rows[
+            r
+        ][1]:
             lt_col_overlap = []
             for c in table.cols:
                 if c[0] <= t.x1 and c[1] >= t.x0:
@@ -561,11 +644,12 @@ def get_table_index(table, t, direction, split_text=False, flag_size=False, stri
                 else:
                     lt_col_overlap.append(-1)
             if len(list(filter(lambda x: x != -1, lt_col_overlap))) == 0:
-                text = t.get_text().strip('\n')
+                text = t.get_text().strip("\n")
                 text_range = (t.x0, t.x1)
                 col_range = (table.cols[0][0], table.cols[-1][1])
-                warnings.warn("{} {} does not lie in column range {}".format(
-                    text, text_range, col_range))
+                warnings.warn(
+                    f"{text} {text_range} does not lie in column range {col_range}"
+                )
             r_idx = r
             c_idx = lt_col_overlap.index(max(lt_col_overlap))
             break
@@ -586,12 +670,26 @@ def get_table_index(table, t, direction, split_text=False, flag_size=False, stri
     error = ((X * (y0_offset + y1_offset)) + (Y * (x0_offset + x1_offset))) / charea
 
     if split_text:
-        return split_textline(table, t, direction, flag_size=flag_size, strip_text=strip_text), error
+        return (
+            split_textline(
+                table, t, direction, flag_size=flag_size, strip_text=strip_text
+            ),
+            error,
+        )
     else:
         if flag_size:
-            return [(r_idx, c_idx, flag_font_size(t._objs, direction, strip_text=strip_text))], error
+            return (
+                [
+                    (
+                        r_idx,
+                        c_idx,
+                        flag_font_size(t._objs, direction, strip_text=strip_text),
+                    )
+                ],
+                error,
+            )
         else:
-            return [(r_idx, c_idx, t.get_text().strip(strip_text))], error
+            return [(r_idx, c_idx, text_strip(t.get_text(), strip_text))], error
 
 
 def compute_accuracy(error_weights):
@@ -642,14 +740,20 @@ def compute_whitespace(d):
     r_nempty_cells, c_nempty_cells = [], []
     for i in d:
         for j in i:
-            if j.strip() == '':
+            if j.strip() == "":
                 whitespace += 1
     whitespace = 100 * (whitespace / float(len(d) * len(d[0])))
     return whitespace
 
 
-def get_page_layout(filename, char_margin=1.0, line_margin=0.5, word_margin=0.1,
-                    detect_vertical=True, all_texts=True):
+def get_page_layout(
+    filename,
+    char_margin=1.0,
+    line_margin=0.5,
+    word_margin=0.1,
+    detect_vertical=True,
+    all_texts=True,
+):
     """Returns a PDFMiner LTPage object and page dimension of a single
     page pdf. See https://euske.github.io/pdfminer/ to get definitions
     of kwargs.
@@ -672,16 +776,18 @@ def get_page_layout(filename, char_margin=1.0, line_margin=0.5, word_margin=0.1,
         Dimension of pdf page in the form (width, height).
 
     """
-    with open(filename, 'rb') as f:
+    with open(filename, "rb") as f:
         parser = PDFParser(f)
         document = PDFDocument(parser)
         if not document.is_extractable:
             raise PDFTextExtractionNotAllowed
-        laparams = LAParams(char_margin=char_margin,
-                            line_margin=line_margin,
-                            word_margin=word_margin,
-                            detect_vertical=detect_vertical,
-                            all_texts=all_texts)
+        laparams = LAParams(
+            char_margin=char_margin,
+            line_margin=line_margin,
+            word_margin=word_margin,
+            detect_vertical=detect_vertical,
+            all_texts=all_texts,
+        )
         rsrcmgr = PDFResourceManager()
         device = PDFPageAggregator(rsrcmgr, laparams=laparams)
         interpreter = PDFPageInterpreter(rsrcmgr, device)
@@ -713,13 +819,13 @@ def get_text_objects(layout, ltype="char", t=None):
         List of PDFMiner text objects.
 
     """
-    if ltype == 'char':
+    if ltype == "char":
         LTObject = LTChar
-    elif ltype == 'image':
+    elif ltype == "image":
         LTObject = LTImage
-    elif ltype == 'horizontal_text':
+    elif ltype == "horizontal_text":
         LTObject = LTTextLineHorizontal
-    elif ltype == 'vertical_text':
+    elif ltype == "vertical_text":
         LTObject = LTTextLineVertical
     if t is None:
         t = []

docs/_templates/sidebarintro.html

@@ -4,13 +4,13 @@
   </a>
 </p>
 <p>
-<iframe src="https://ghbtns.com/github-btn.html?user=socialcopsdev&repo=camelot&type=watch&count=true&size=large"
+<iframe src="https://ghbtns.com/github-btn.html?user=camelot-dev&repo=camelot&type=watch&count=true&size=large"
   allowtransparency="true" frameborder="0" scrolling="0" width="200px" height="35px"></iframe>
 </p>
 
 <h3>Useful Links</h3>
 <ul>
-  <li><a href="https://github.com/socialcopsdev/camelot">Camelot @ GitHub</a></li>
+  <li><a href="https://github.com/camelot-dev/camelot">Camelot @ GitHub</a></li>
   <li><a href="https://pypi.org/project/camelot-py/">Camelot @ PyPI</a></li>
-  <li><a href="https://github.com/socialcopsdev/camelot/issues">Issue Tracker</a></li>
-</ul>
+  <li><a href="https://github.com/camelot-dev/camelot/issues">Issue Tracker</a></li>
+</ul>

docs/_templates/sidebarlogo.html

@@ -4,6 +4,6 @@
   </a>
 </p>
 <p>
-<iframe src="https://ghbtns.com/github-btn.html?user=socialcopsdev&repo=camelot&type=watch&count=true&size=large"
+<iframe src="https://ghbtns.com/github-btn.html?user=camelot-dev&repo=camelot&type=watch&count=true&size=large"
   allowtransparency="true" frameborder="0" scrolling="0" width="200px" height="35px"></iframe>
-</p>
+</p>

docs/conf.py

@@ -63,7 +63,7 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'Camelot'
-copyright = u'2018, <a href="https://socialcops.com" target="_blank">SocialCops</a>'
+copyright = u'2019, Camelot Developers'
 author = u'Vinayak Mehta'
 
 # The version info for the project you're documenting, acts as replacement for
@@ -137,7 +137,7 @@ html_theme = 'alabaster'
 # documentation.
 html_theme_options = {
     'show_powered_by': False,
-    'github_user': 'socialcopsdev',
+    'github_user': 'camelot-dev',
     'github_repo': 'camelot',
     'github_banner': True,
     'show_related': False,

docs/dev/contributing.rst

@@ -29,8 +29,8 @@ Your first contribution
 
 A great way to start contributing to Camelot is to pick an issue tagged with the `help wanted`_ or the `good first issue`_ tags. If you're unable to find a good first issue, feel free to contact the maintainer.
 
-.. _help wanted: https://github.com/socialcopsdev/camelot/labels/help%20wanted
-.. _good first issue: https://github.com/socialcopsdev/camelot/labels/good%20first%20issue
+.. _help wanted: https://github.com/camelot-dev/camelot/labels/help%20wanted
+.. _good first issue: https://github.com/camelot-dev/camelot/labels/good%20first%20issue
 
 Setting up a development environment
 ------------------------------------
@@ -51,7 +51,7 @@ Submit a pull request
 
 The preferred workflow for contributing to Camelot is to fork the `project repository`_ on GitHub, clone, develop on a branch and then finally submit a pull request. Here are the steps:
 
-.. _project repository: https://github.com/socialcopsdev/camelot
+.. _project repository: https://github.com/camelot-dev/camelot
 
 1. Fork the project repository. Click on the ‘Fork’ button near the top of the page. This creates a copy of the code under your account on the GitHub.
 
@@ -134,7 +134,7 @@ Filing Issues
 
 We use `GitHub issues`_ to keep track of all issues and pull requests. Before opening an issue (which asks a question or reports a bug), please use GitHub search to look for existing issues (both open and closed) that may be similar.
 
-.. _GitHub issues: https://github.com/socialcopsdev/camelot/issues
+.. _GitHub issues: https://github.com/camelot-dev/camelot/issues
 
 Questions
 ^^^^^^^^^

docs/index.rst

@@ -8,15 +8,15 @@ Camelot: PDF Table Extraction for Humans
 
 Release v\ |version|. (:ref:`Installation <install>`)
 
-.. image:: https://travis-ci.org/socialcopsdev/camelot.svg?branch=master
-    :target: https://travis-ci.org/socialcopsdev/camelot
+.. image:: https://travis-ci.org/camelot-dev/camelot.svg?branch=master
+    :target: https://travis-ci.org/camelot-dev/camelot
 
 .. image:: https://readthedocs.org/projects/camelot-py/badge/?version=master
     :target: https://camelot-py.readthedocs.io/en/master/
     :alt: Documentation Status
 
-.. image:: https://codecov.io/github/socialcopsdev/camelot/badge.svg?branch=master&service=github
-    :target: https://codecov.io/github/socialcopsdev/camelot?branch=master
+.. image:: https://codecov.io/github/camelot-dev/camelot/badge.svg?branch=master&service=github
+    :target: https://codecov.io/github/camelot-dev/camelot?branch=master
 
 .. image:: https://img.shields.io/pypi/v/camelot-py.svg
     :target: https://pypi.org/project/camelot-py/
@@ -30,6 +30,12 @@ Release v\ |version|. (:ref:`Installation <install>`)
 .. image:: https://badges.gitter.im/camelot-dev/Lobby.png
     :target: https://gitter.im/camelot-dev/Lobby
 
+.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
+    :target: https://github.com/ambv/black
+
+.. image:: https://img.shields.io/badge/continous%20quality-deepsource-lightgrey
+    :target: https://deepsource.io/gh/camelot-dev/camelot/?ref=repository-badge
+
 **Camelot** is a Python library that makes it easy for *anyone* to extract tables from PDF files!
 
 .. note:: You can also check out `Excalibur`_, which is a web interface for Camelot!
@@ -81,7 +87,14 @@ Why Camelot?
 See `comparison with other PDF table extraction libraries and tools`_.
 
 .. _ETL and data analysis workflows: https://gist.github.com/vinayak-mehta/e5949f7c2410a0e12f25d3682dc9e873
-.. _comparison with other PDF table extraction libraries and tools: https://github.com/socialcopsdev/camelot/wiki/Comparison-with-other-PDF-Table-Extraction-libraries-and-tools
+.. _comparison with other PDF table extraction libraries and tools: https://github.com/camelot-dev/camelot/wiki/Comparison-with-other-PDF-Table-Extraction-libraries-and-tools
+
+Support us on OpenCollective
+----------------------------
+
+If Camelot helped you extract tables from PDFs, please consider supporting its development by `becoming a backer or a sponsor on OpenCollective`_!
+
+.. _becoming a backer or a sponsor on OpenCollective: https://opencollective.com/camelot
 
 The User Guide
 --------------

docs/user/advanced.rst

@@ -224,10 +224,12 @@ Table areas that you want Camelot to analyze can be passed as a list of comma-se
 .. csv-table::
   :file: ../_static/csv/table_areas.csv
 
+.. note:: ``table_areas`` accepts strings of the form x1,y1,x2,y2 where (x1, y1) -> top-left and (x2, y2) -> bottom-right in PDF coordinate space. In PDF coordinate space, the bottom-left corner of the page is the origin, with coordinates (0, 0).
+
 Specify table regions
 ---------------------
 
-However there may be cases like `[1] <../_static/pdf/table_regions.pdf>`__ and `[2] <https://github.com/socialcopsdev/camelot/blob/master/tests/files/tableception.pdf>`__, where the table might not lie at the exact coordinates every time but in an approximate region.
+However there may be cases like `[1] <../_static/pdf/table_regions.pdf>`__ and `[2] <https://github.com/camelot-dev/camelot/blob/master/tests/files/tableception.pdf>`__, where the table might not lie at the exact coordinates every time but in an approximate region.
 
 You can use the ``table_regions`` keyword argument to :meth:`read_pdf() <camelot.read_pdf>` to solve for such cases. When ``table_regions`` is specified, Camelot will only analyze the specified regions to look for tables.
 
@@ -314,7 +316,7 @@ In this case, the text that `other tools`_ return, will be ``24.912``. This is r
 
 You can solve this by passing ``flag_size=True``, which will enclose the superscripts and subscripts with ``<s></s>``, based on font size, as shown below.
 
-.. _other tools: https://github.com/socialcopsdev/camelot/wiki/Comparison-with-other-PDF-Table-Extraction-libraries-and-tools
+.. _other tools: https://github.com/camelot-dev/camelot/wiki/Comparison-with-other-PDF-Table-Extraction-libraries-and-tools
 
 ::
 
@@ -338,7 +340,7 @@ You can solve this by passing ``flag_size=True``, which will enclose the supersc
 Strip characters from text
 --------------------------
 
-You can strip unwanted characters like spaces, dots and newlines from a string using the ``strip_text`` keyword argument. Take a look at `this PDF <https://github.com/socialcopsdev/camelot/blob/master/tests/files/tabula/12s0324.pdf>`_ as an example, the text at the start of each row contains a lot of unwanted spaces, dots and newlines.
+You can strip unwanted characters like spaces, dots and newlines from a string using the ``strip_text`` keyword argument. Take a look at `this PDF <https://github.com/camelot-dev/camelot/blob/master/tests/files/tabula/12s0324.pdf>`_ as an example, the text at the start of each row contains a lot of unwanted spaces, dots and newlines.
 
 ::
 
@@ -364,7 +366,7 @@ You can strip unwanted characters like spaces, dots and newlines from a string u
 Improve guessed table areas
 ---------------------------
 
-While using :ref:`Stream <stream>`, automatic table detection can fail for PDFs like `this one <https://github.com/socialcopsdev/camelot/blob/master/tests/files/edge_tol.pdf>`_. That's because the text is relatively far apart vertically, which can lead to shorter textedges being calculated.
+While using :ref:`Stream <stream>`, automatic table detection can fail for PDFs like `this one <https://github.com/camelot-dev/camelot/blob/master/tests/files/edge_tol.pdf>`_. That's because the text is relatively far apart vertically, which can lead to shorter textedges being calculated.
 
 .. note:: To know more about how textedges are calculated to guess table areas, you can see pages 20, 35 and 40 of `Anssi Nurminen's master's thesis <http://dspace.cc.tut.fi/dpub/bitstream/handle/123456789/21520/Nurminen.pdf?sequence=3>`_.
 
@@ -624,7 +626,7 @@ We don't need anything else. Now, let's pass ``copy_text=['v']`` to copy text in
 Tweak layout generation
 -----------------------
 
-Camelot is built on top of PDFMiner's functionality of grouping characters on a page into words and sentences. In some cases (such as `#170 <https://github.com/socialcopsdev/camelot/issues/170>`_ and `#215 <https://github.com/socialcopsdev/camelot/issues/215>`_), PDFMiner can group characters that should belong to the same sentence into separate sentences.
+Camelot is built on top of PDFMiner's functionality of grouping characters on a page into words and sentences. In some cases (such as `#170 <https://github.com/camelot-dev/camelot/issues/170>`_ and `#215 <https://github.com/camelot-dev/camelot/issues/215>`_), PDFMiner can group characters that should belong to the same sentence into separate sentences.
 
 To deal with such cases, you can tweak PDFMiner's `LAParams kwargs <https://github.com/euske/pdfminer/blob/master/pdfminer/layout.py#L33>`_ to improve layout generation, by passing the keyword arguments as a dict using ``layout_kwargs`` in :meth:`read_pdf() <camelot.read_pdf>`. To know more about the parameters you can tweak, you can check out `PDFMiner docs <https://euske.github.io/pdfminer/>`_.
 

docs/user/cli.rst

@@ -26,6 +26,8 @@ You can print the help for the interface by typing ``camelot --help`` in your fa
     -split, --split_text            Split text that spans across multiple cells.
     -flag, --flag_size              Flag text based on font size. Useful to
                                     detect super/subscripts.
+    -strip, --strip_text            Characters that should be stripped from a
+                                    string before assigning it to a cell.
     -M, --margins <FLOAT FLOAT FLOAT>...
                                     PDFMiner char_margin, line_margin and
                                     word_margin.

docs/user/how-it-works.rst

@@ -20,7 +20,7 @@ Stream can be used to parse tables that have whitespaces between cells to simula
 
 3. The number of columns inside each table area are then guessed. This is done by calculating the mode of number of words in each text row. Based on this mode, words in each text row are chosen to calculate a list of column *x* ranges.
 
-4. Words that lie inside/outside the current column *x* ranges are then used to extend extend the current list of columns.
+4. Words that lie inside/outside the current column *x* ranges are then used to extend the current list of columns.
 
 5. Finally, a table is formed using the text rows' *y* ranges and column *x* ranges and words found on the page are assigned to the table's cells based on their *x* and *y* coordinates.
 

docs/user/install.rst

@@ -13,7 +13,7 @@ The easiest way to install Camelot is to install it with `conda`_, which is a pa
 
     $ conda install -c conda-forge camelot-py
 
-.. note:: Camelot is available for Python 2.7, 3.5 and 3.6 on Linux, macOS and Windows. For Windows, you will need to install ghostscript which you can get from their `downloads page`_.
+.. note:: Camelot is available for Python 2.7, 3.5, 3.6 and 3.7 on Linux, macOS and Windows. For Windows, you will need to install ghostscript which you can get from their `downloads page`_.
 
 .. _conda: https://conda.io/docs/
 .. _Anaconda: http://docs.continuum.io/anaconda/
@@ -38,7 +38,7 @@ After :ref:`installing the dependencies <install_deps>`, you can install from th
 1. Cloning the GitHub repository.
 ::
 
-    $ git clone https://www.github.com/socialcopsdev/camelot
+    $ git clone https://www.github.com/camelot-dev/camelot
 
 2. Then simply using pip again.
 ::

docs/user/intro.rst

@@ -27,7 +27,7 @@ Here is a `comparison`_ of Camelot's output with outputs from other open-source
 .. _pdf-table-extract: https://github.com/ashima/pdf-table-extract
 .. _PDFTables: https://pdftables.com/
 .. _Smallpdf: https://smallpdf.com
-.. _comparison: https://github.com/socialcopsdev/camelot/wiki/Comparison-with-other-PDF-Table-Extraction-libraries-and-tools
+.. _comparison: https://github.com/camelot-dev/camelot/wiki/Comparison-with-other-PDF-Table-Extraction-libraries-and-tools
 
 What's in a name?
 -----------------

setup.py

@@ -71,10 +71,9 @@ def setup_package():
                         # Trove classifiers
                         # Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers
                         'License :: OSI Approved :: MIT License',
-                        'Programming Language :: Python :: 2.7',
-                        'Programming Language :: Python :: 3.5',
                         'Programming Language :: Python :: 3.6',
-                        'Programming Language :: Python :: 3.7'
+                        'Programming Language :: Python :: 3.7',
+                        'Programming Language :: Python :: 3.8'
                     ])
 
     try:

tests/__init__.py

@@ -1,2 +1,3 @@
 import matplotlib
-matplotlib.use('agg')
+
+matplotlib.use("agg")

tests/test_cli.py

@@ -9,109 +9,167 @@ from camelot.utils import TemporaryDirectory
 
 
 testdir = os.path.dirname(os.path.abspath(__file__))
-testdir = os.path.join(testdir, 'files')
+testdir = os.path.join(testdir, "files")
+
+
+def test_help_output():
+    runner = CliRunner()
+    prog_name = runner.get_default_prog_name(cli)
+    result = runner.invoke(cli, ["--help"])
+    output = result.output
+
+    assert prog_name == "camelot"
+    assert result.output.startswith("Usage: %(prog_name)s [OPTIONS] COMMAND" % locals())
+    assert all(
+        v in result.output
+        for v in ["Options:", "--version", "--help", "Commands:", "lattice", "stream"]
+    )
 
 
 def test_cli_lattice():
     with TemporaryDirectory() as tempdir:
-        infile = os.path.join(testdir, 'foo.pdf')
-        outfile = os.path.join(tempdir, 'foo.csv')
+        infile = os.path.join(testdir, "foo.pdf")
+        outfile = os.path.join(tempdir, "foo.csv")
         runner = CliRunner()
-        result = runner.invoke(cli, ['--format', 'csv', '--output', outfile,
-                                     'lattice', infile])
+        result = runner.invoke(
+            cli, ["--format", "csv", "--output", outfile, "lattice", infile]
+        )
         assert result.exit_code == 0
-        assert result.output == 'Found 1 tables\n'
+        assert result.output == "Found 1 tables\n"
 
-        result = runner.invoke(cli, ['--format', 'csv',
-                                     'lattice', infile])
-        output_error = 'Error: Please specify output file path using --output'
+        result = runner.invoke(cli, ["--format", "csv", "lattice", infile])
+        output_error = "Error: Please specify output file path using --output"
         assert output_error in result.output
 
-        result = runner.invoke(cli, ['--output', outfile,
-                                     'lattice', infile])
-        format_error = 'Please specify output file format using --format'
+        result = runner.invoke(cli, ["--output", outfile, "lattice", infile])
+        format_error = "Please specify output file format using --format"
         assert format_error in result.output
 
 
 def test_cli_stream():
     with TemporaryDirectory() as tempdir:
-        infile = os.path.join(testdir, 'budget.pdf')
-        outfile = os.path.join(tempdir, 'budget.csv')
+        infile = os.path.join(testdir, "budget.pdf")
+        outfile = os.path.join(tempdir, "budget.csv")
         runner = CliRunner()
-        result = runner.invoke(cli, ['--format', 'csv', '--output', outfile,
-                                     'stream', infile])
+        result = runner.invoke(
+            cli, ["--format", "csv", "--output", outfile, "stream", infile]
+        )
         assert result.exit_code == 0
-        assert result.output == 'Found 1 tables\n'
+        assert result.output == "Found 1 tables\n"
 
-        result = runner.invoke(cli, ['--format', 'csv', 'stream', infile])
-        output_error = 'Error: Please specify output file path using --output'
+        result = runner.invoke(cli, ["--format", "csv", "stream", infile])
+        output_error = "Error: Please specify output file path using --output"
         assert output_error in result.output
 
-        result = runner.invoke(cli, ['--output', outfile, 'stream', infile])
-        format_error = 'Please specify output file format using --format'
+        result = runner.invoke(cli, ["--output", outfile, "stream", infile])
+        format_error = "Please specify output file format using --format"
         assert format_error in result.output
 
 
 def test_cli_password():
     with TemporaryDirectory() as tempdir:
-        infile = os.path.join(testdir, 'health_protected.pdf')
-        outfile = os.path.join(tempdir, 'health_protected.csv')
+        infile = os.path.join(testdir, "health_protected.pdf")
+        outfile = os.path.join(tempdir, "health_protected.csv")
         runner = CliRunner()
-        result = runner.invoke(cli, ['--password', 'userpass',
-                                     '--format', 'csv', '--output', outfile,
-                                     'stream', infile])
+        result = runner.invoke(
+            cli,
+            [
+                "--password",
+                "userpass",
+                "--format",
+                "csv",
+                "--output",
+                outfile,
+                "stream",
+                infile,
+            ],
+        )
         assert result.exit_code == 0
-        assert result.output == 'Found 1 tables\n'
+        assert result.output == "Found 1 tables\n"
 
-        output_error = 'file has not been decrypted'
+        output_error = "file has not been decrypted"
         # no password
-        result = runner.invoke(cli, ['--format', 'csv', '--output', outfile,
-                                     'stream', infile])
+        result = runner.invoke(
+            cli, ["--format", "csv", "--output", outfile, "stream", infile]
+        )
         assert output_error in str(result.exception)
 
         # bad password
-        result = runner.invoke(cli, ['--password', 'wrongpass',
-                                     '--format', 'csv', '--output', outfile,
-                                     'stream', infile])
+        result = runner.invoke(
+            cli,
+            [
+                "--password",
+                "wrongpass",
+                "--format",
+                "csv",
+                "--output",
+                outfile,
+                "stream",
+                infile,
+            ],
+        )
         assert output_error in str(result.exception)
 
 
 def test_cli_output_format():
     with TemporaryDirectory() as tempdir:
-        infile = os.path.join(testdir, 'health.pdf')
-        outfile = os.path.join(tempdir, 'health.{}')
+        infile = os.path.join(testdir, "health.pdf")
+
         runner = CliRunner()
 
         # json
-        result = runner.invoke(cli, ['--format', 'json', '--output', outfile.format('json'),
-                                     'stream', infile])
+        outfile = os.path.join(tempdir, "health.json")
+        result = runner.invoke(
+            cli,
+            ["--format", "json", "--output", outfile, "stream", infile],
+        )
         assert result.exit_code == 0
 
         # excel
-        result = runner.invoke(cli, ['--format', 'excel', '--output', outfile.format('xlsx'),
-                                     'stream', infile])
+        outfile = os.path.join(tempdir, "health.xlsx")
+        result = runner.invoke(
+            cli,
+            ["--format", "excel", "--output", outfile, "stream", infile],
+        )
         assert result.exit_code == 0
 
         # html
-        result = runner.invoke(cli, ['--format', 'html', '--output', outfile.format('html'),
-                                     'stream', infile])
+        outfile = os.path.join(tempdir, "health.html")
+        result = runner.invoke(
+            cli,
+            ["--format", "html", "--output", outfile, "stream", infile],
+        )
         assert result.exit_code == 0
 
         # zip
-        result = runner.invoke(cli, ['--zip', '--format', 'csv', '--output', outfile.format('csv'),
-                                     'stream', infile])
+        outfile = os.path.join(tempdir, "health.csv")
+        result = runner.invoke(
+            cli,
+            [
+                "--zip",
+                "--format",
+                "csv",
+                "--output",
+                outfile,
+                "stream",
+                infile,
+            ],
+        )
         assert result.exit_code == 0
 
+
 def test_cli_quiet():
     with TemporaryDirectory() as tempdir:
-        infile = os.path.join(testdir, 'blank.pdf')
-        outfile = os.path.join(tempdir, 'blank.csv')
+        infile = os.path.join(testdir, "blank.pdf")
+        outfile = os.path.join(tempdir, "blank.csv")
         runner = CliRunner()
 
-        result = runner.invoke(cli, ['--format', 'csv', '--output', outfile,
-                                     'stream', infile])
-        assert 'No tables found on page-1' in result.output
+        result = runner.invoke(
+            cli, ["--format", "csv", "--output", outfile, "stream", infile]
+        )
+        assert "No tables found on page-1" in result.output
 
-        result = runner.invoke(cli, ['--quiet', '--format', 'csv',
-                                     '--output', outfile, 'stream', infile])
-        assert 'No tables found on page-1' not in result.output
+        result = runner.invoke(
+            cli, ["--quiet", "--format", "csv", "--output", outfile, "stream", infile]
+        )
+        assert "No tables found on page-1" not in result.output

tests/test_common.py

@@ -3,8 +3,11 @@
 import os
 
 import pandas as pd
+from pandas.testing import assert_frame_equal
 
 import camelot
+from camelot.core import Table, TableList
+from camelot.__version__ import generate_version
 
 from .data import *
 
@@ -13,12 +16,7 @@ testdir = os.path.join(testdir, "files")
 
 
 def test_parsing_report():
-    parsing_report = {
-        'accuracy': 99.02,
-        'whitespace': 12.24,
-        'order': 1,
-        'page': 1
-    }
+    parsing_report = {"accuracy": 99.02, "whitespace": 12.24, "order": 1, "page": 1}
 
     filename = os.path.join(testdir, "foo.pdf")
     tables = camelot.read_pdf(filename)
@@ -30,10 +28,10 @@ def test_password():
 
     filename = os.path.join(testdir, "health_protected.pdf")
     tables = camelot.read_pdf(filename, password="ownerpass", flavor="stream")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
     tables = camelot.read_pdf(filename, password="userpass", flavor="stream")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream():
@@ -41,7 +39,7 @@ def test_stream():
 
     filename = os.path.join(testdir, "health.pdf")
     tables = camelot.read_pdf(filename, flavor="stream")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_table_rotated():
@@ -49,11 +47,11 @@ def test_stream_table_rotated():
 
     filename = os.path.join(testdir, "clockwise_table_2.pdf")
     tables = camelot.read_pdf(filename, flavor="stream")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
     filename = os.path.join(testdir, "anticlockwise_table_2.pdf")
     tables = camelot.read_pdf(filename, flavor="stream")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_two_tables():
@@ -61,19 +59,31 @@ def test_stream_two_tables():
     df2 = pd.DataFrame(data_stream_two_tables_2)
 
     filename = os.path.join(testdir, "tabula/12s0324.pdf")
-    tables = camelot.read_pdf(filename, flavor='stream')
+    tables = camelot.read_pdf(filename, flavor="stream")
 
     assert len(tables) == 2
     assert df1.equals(tables[0].df)
     assert df2.equals(tables[1].df)
 
 
+def test_stream_table_regions():
+    df = pd.DataFrame(data_stream_table_areas)
+
+    filename = os.path.join(testdir, "tabula/us-007.pdf")
+    tables = camelot.read_pdf(
+        filename, flavor="stream", table_regions=["320,460,573,335"]
+    )
+    assert_frame_equal(df, tables[0].df)
+
+
 def test_stream_table_areas():
     df = pd.DataFrame(data_stream_table_areas)
 
     filename = os.path.join(testdir, "tabula/us-007.pdf")
-    tables = camelot.read_pdf(filename, flavor="stream", table_areas=["320,500,573,335"])
-    assert df.equals(tables[0].df)
+    tables = camelot.read_pdf(
+        filename, flavor="stream", table_areas=["320,500,573,335"]
+    )
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_columns():
@@ -81,8 +91,9 @@ def test_stream_columns():
 
     filename = os.path.join(testdir, "mexican_towns.pdf")
     tables = camelot.read_pdf(
-        filename, flavor="stream", columns=["67,180,230,425,475"], row_tol=10)
-    assert df.equals(tables[0].df)
+        filename, flavor="stream", columns=["67,180,230,425,475"], row_tol=10
+    )
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_split_text():
@@ -90,8 +101,12 @@ def test_stream_split_text():
 
     filename = os.path.join(testdir, "tabula/m27.pdf")
     tables = camelot.read_pdf(
-        filename, flavor="stream", columns=["72,95,209,327,442,529,566,606,683"], split_text=True)
-    assert df.equals(tables[0].df)
+        filename,
+        flavor="stream",
+        columns=["72,95,209,327,442,529,566,606,683"],
+        split_text=True,
+    )
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_flag_size():
@@ -99,15 +114,15 @@ def test_stream_flag_size():
 
     filename = os.path.join(testdir, "superscript.pdf")
     tables = camelot.read_pdf(filename, flavor="stream", flag_size=True)
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_strip_text():
     df = pd.DataFrame(data_stream_strip_text)
 
     filename = os.path.join(testdir, "detect_vertical_false.pdf")
-    tables = camelot.read_pdf(filename, flavor="stream", strip_text="\n")
-    assert df.equals(tables[0].df)
+    tables = camelot.read_pdf(filename, flavor="stream", strip_text=" ,\n")
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_edge_tol():
@@ -115,7 +130,7 @@ def test_stream_edge_tol():
 
     filename = os.path.join(testdir, "edge_tol.pdf")
     tables = camelot.read_pdf(filename, flavor="stream", edge_tol=500)
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_layout_kwargs():
@@ -123,17 +138,19 @@ def test_stream_layout_kwargs():
 
     filename = os.path.join(testdir, "detect_vertical_false.pdf")
     tables = camelot.read_pdf(
-        filename, flavor="stream", layout_kwargs={"detect_vertical": False})
-    assert df.equals(tables[0].df)
+        filename, flavor="stream", layout_kwargs={"detect_vertical": False}
+    )
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_lattice():
     df = pd.DataFrame(data_lattice)
 
     filename = os.path.join(
-        testdir, "tabula/icdar2013-dataset/competition-dataset-us/us-030.pdf")
+        testdir, "tabula/icdar2013-dataset/competition-dataset-us/us-030.pdf"
+    )
     tables = camelot.read_pdf(filename, pages="2")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_lattice_table_rotated():
@@ -141,11 +158,11 @@ def test_lattice_table_rotated():
 
     filename = os.path.join(testdir, "clockwise_table_1.pdf")
     tables = camelot.read_pdf(filename)
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
     filename = os.path.join(testdir, "anticlockwise_table_1.pdf")
     tables = camelot.read_pdf(filename)
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_lattice_two_tables():
@@ -164,7 +181,7 @@ def test_lattice_table_regions():
 
     filename = os.path.join(testdir, "table_region.pdf")
     tables = camelot.read_pdf(filename, table_regions=["170,370,560,270"])
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_lattice_table_areas():
@@ -172,7 +189,7 @@ def test_lattice_table_areas():
 
     filename = os.path.join(testdir, "twotables_2.pdf")
     tables = camelot.read_pdf(filename, table_areas=["80,693,535,448"])
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_lattice_process_background():
@@ -180,7 +197,7 @@ def test_lattice_process_background():
 
     filename = os.path.join(testdir, "background_lines_1.pdf")
     tables = camelot.read_pdf(filename, process_background=True)
-    assert df.equals(tables[1].df)
+    assert_frame_equal(df, tables[1].df)
 
 
 def test_lattice_copy_text():
@@ -188,7 +205,7 @@ def test_lattice_copy_text():
 
     filename = os.path.join(testdir, "row_span_1.pdf")
     tables = camelot.read_pdf(filename, line_scale=60, copy_text="v")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_lattice_shift_text():
@@ -200,10 +217,10 @@ def test_lattice_shift_text():
     tables = camelot.read_pdf(filename, line_scale=40)
     assert df_lt.equals(tables[0].df)
 
-    tables = camelot.read_pdf(filename, line_scale=40, shift_text=[''])
+    tables = camelot.read_pdf(filename, line_scale=40, shift_text=[""])
     assert df_disable.equals(tables[0].df)
 
-    tables = camelot.read_pdf(filename, line_scale=40, shift_text=['r', 'b'])
+    tables = camelot.read_pdf(filename, line_scale=40, shift_text=["r", "b"])
     assert df_rb.equals(tables[0].df)
 
 
@@ -212,7 +229,9 @@ def test_repr():
     tables = camelot.read_pdf(filename)
     assert repr(tables) == "<TableList n=1>"
     assert repr(tables[0]) == "<Table shape=(7, 7)>"
-    assert repr(tables[0].cells[0][0]) == "<Cell x1=120.48 y1=218.43 x2=164.64 y2=233.77>"
+    assert (
+        repr(tables[0].cells[0][0]) == "<Cell x1=120.48 y1=218.43 x2=164.64 y2=233.77>"
+    )
 
 
 def test_pages():
@@ -220,17 +239,23 @@ def test_pages():
     tables = camelot.read_pdf(url)
     assert repr(tables) == "<TableList n=1>"
     assert repr(tables[0]) == "<Table shape=(7, 7)>"
-    assert repr(tables[0].cells[0][0]) == "<Cell x1=120.48 y1=218.43 x2=164.64 y2=233.77>"
+    assert (
+        repr(tables[0].cells[0][0]) == "<Cell x1=120.48 y1=218.43 x2=164.64 y2=233.77>"
+    )
 
-    tables = camelot.read_pdf(url, pages='1-end')
+    tables = camelot.read_pdf(url, pages="1-end")
     assert repr(tables) == "<TableList n=1>"
     assert repr(tables[0]) == "<Table shape=(7, 7)>"
-    assert repr(tables[0].cells[0][0]) == "<Cell x1=120.48 y1=218.43 x2=164.64 y2=233.77>"
+    assert (
+        repr(tables[0].cells[0][0]) == "<Cell x1=120.48 y1=218.43 x2=164.64 y2=233.77>"
+    )
 
-    tables = camelot.read_pdf(url, pages='all')
+    tables = camelot.read_pdf(url, pages="all")
     assert repr(tables) == "<TableList n=1>"
     assert repr(tables[0]) == "<Table shape=(7, 7)>"
-    assert repr(tables[0].cells[0][0]) == "<Cell x1=120.48 y1=218.43 x2=164.64 y2=233.77>"
+    assert (
+        repr(tables[0].cells[0][0]) == "<Cell x1=120.48 y1=218.43 x2=164.64 y2=233.77>"
+    )
 
 
 def test_url():
@@ -238,7 +263,9 @@ def test_url():
     tables = camelot.read_pdf(url)
     assert repr(tables) == "<TableList n=1>"
     assert repr(tables[0]) == "<Table shape=(7, 7)>"
-    assert repr(tables[0].cells[0][0]) == "<Cell x1=120.48 y1=218.43 x2=164.64 y2=233.77>"
+    assert (
+        repr(tables[0].cells[0][0]) == "<Cell x1=120.48 y1=218.43 x2=164.64 y2=233.77>"
+    )
 
 
 def test_arabic():
@@ -246,4 +273,44 @@ def test_arabic():
 
     filename = os.path.join(testdir, "tabula/arabic.pdf")
     tables = camelot.read_pdf(filename)
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
+
+
+def test_table_order():
+    def _make_table(page, order):
+        t = Table([], [])
+        t.page = page
+        t.order = order
+        return t
+
+    table_list = TableList(
+        [_make_table(2, 1), _make_table(1, 1), _make_table(3, 4), _make_table(1, 2)]
+    )
+
+    assert [(t.page, t.order) for t in sorted(table_list)] == [
+        (1, 1),
+        (1, 2),
+        (2, 1),
+        (3, 4),
+    ]
+    assert [(t.page, t.order) for t in sorted(table_list, reverse=True)] == [
+        (3, 4),
+        (2, 1),
+        (1, 2),
+        (1, 1),
+    ]
+
+
+def test_version_generation():
+    version = (0, 7, 3)
+    assert generate_version(version, prerelease=None, revision=None) == "0.7.3"
+
+
+def test_version_generation_with_prerelease_revision():
+    version = (0, 7, 3)
+    prerelease = "alpha"
+    revision = 2
+    assert (
+        generate_version(version, prerelease=prerelease, revision=revision)
+        == "0.7.3-alpha.2"
+    )

tests/test_errors.py

@@ -10,88 +10,93 @@ import camelot
 
 testdir = os.path.dirname(os.path.abspath(__file__))
 testdir = os.path.join(testdir, "files")
-filename = os.path.join(testdir, 'foo.pdf')
+filename = os.path.join(testdir, "foo.pdf")
 
 
 def test_unknown_flavor():
-    message = ("Unknown flavor specified."
-               " Use either 'lattice' or 'stream'")
-    with pytest.raises(NotImplementedError, message=message):
-        tables = camelot.read_pdf(filename, flavor='chocolate')
+    message = "Unknown flavor specified." " Use either 'lattice' or 'stream'"
+    with pytest.raises(NotImplementedError, match=message):
+        tables = camelot.read_pdf(filename, flavor="chocolate")
 
 
 def test_input_kwargs():
     message = "columns cannot be used with flavor='lattice'"
-    with pytest.raises(ValueError, message=message):
-        tables = camelot.read_pdf(filename, columns=['10,20,30,40'])
+    with pytest.raises(ValueError, match=message):
+        tables = camelot.read_pdf(filename, columns=["10,20,30,40"])
 
 
 def test_unsupported_format():
-    message = 'File format not supported'
-    filename = os.path.join(testdir, 'foo.csv')
-    with pytest.raises(NotImplementedError, message=message):
+    message = "File format not supported"
+    filename = os.path.join(testdir, "foo.csv")
+    with pytest.raises(NotImplementedError, match=message):
         tables = camelot.read_pdf(filename)
 
 
 def test_stream_equal_length():
-    message = ("Length of table_areas and columns"
-               " should be equal")
-    with pytest.raises(ValueError, message=message):
-        tables = camelot.read_pdf(filename, flavor='stream',
-            table_areas=['10,20,30,40'], columns=['10,20,30,40', '10,20,30,40'])
+    message = "Length of table_areas and columns" " should be equal"
+    with pytest.raises(ValueError, match=message):
+        tables = camelot.read_pdf(
+            filename,
+            flavor="stream",
+            table_areas=["10,20,30,40"],
+            columns=["10,20,30,40", "10,20,30,40"],
+        )
 
 
 def test_image_warning():
-    filename = os.path.join(testdir, 'image.pdf')
+    filename = os.path.join(testdir, "image.pdf")
     with warnings.catch_warnings():
-        warnings.simplefilter('error')
+        warnings.simplefilter("error")
         with pytest.raises(UserWarning) as e:
             tables = camelot.read_pdf(filename)
-        assert str(e.value) == 'page-1 is image-based, camelot only works on text-based pages.'
+            assert (
+                str(e.value)
+                == "page-1 is image-based, camelot only works on text-based pages."
+            )
 
 
 def test_no_tables_found():
-    filename = os.path.join(testdir, 'blank.pdf')
+    filename = os.path.join(testdir, "blank.pdf")
     with warnings.catch_warnings():
-        warnings.simplefilter('error')
+        warnings.simplefilter("error")
         with pytest.raises(UserWarning) as e:
             tables = camelot.read_pdf(filename)
-        assert str(e.value) == 'No tables found on page-1'
+        assert str(e.value) == "No tables found on page-1"
 
 
 def test_no_tables_found_logs_suppressed():
-    filename = os.path.join(testdir, 'foo.pdf')
+    filename = os.path.join(testdir, "foo.pdf")
     with warnings.catch_warnings():
         # the test should fail if any warning is thrown
-        warnings.simplefilter('error')
+        warnings.simplefilter("error")
         try:
             tables = camelot.read_pdf(filename, suppress_stdout=True)
         except Warning as e:
             warning_text = str(e)
-            pytest.fail('Unexpected warning: {}'.format(warning_text))
+            pytest.fail(f"Unexpected warning: {warning_text}")
 
 
 def test_no_tables_found_warnings_suppressed():
-    filename = os.path.join(testdir, 'blank.pdf')
+    filename = os.path.join(testdir, "blank.pdf")
     with warnings.catch_warnings():
         # the test should fail if any warning is thrown
-        warnings.simplefilter('error')
+        warnings.simplefilter("error")
         try:
             tables = camelot.read_pdf(filename, suppress_stdout=True)
         except Warning as e:
             warning_text = str(e)
-            pytest.fail('Unexpected warning: {}'.format(warning_text))
+            pytest.fail(f"Unexpected warning: {warning_text}")
 
 
 def test_no_password():
-    filename = os.path.join(testdir, 'health_protected.pdf')
-    message = 'file has not been decrypted'
-    with pytest.raises(Exception, message=message):
+    filename = os.path.join(testdir, "health_protected.pdf")
+    message = "file has not been decrypted"
+    with pytest.raises(Exception, match=message):
         tables = camelot.read_pdf(filename)
 
 
 def test_bad_password():
-    filename = os.path.join(testdir, 'health_protected.pdf')
-    message = 'file has not been decrypted'
-    with pytest.raises(Exception, message=message):
-        tables = camelot.read_pdf(filename, password='wrongpass')
+    filename = os.path.join(testdir, "health_protected.pdf")
+    message = "file has not been decrypted"
+    with pytest.raises(Exception, match=message):
+        tables = camelot.read_pdf(filename, password="wrongpass")

tests/test_plotting.py

@@ -11,57 +11,50 @@ testdir = os.path.dirname(os.path.abspath(__file__))
 testdir = os.path.join(testdir, "files")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_text_plot():
     filename = os.path.join(testdir, "foo.pdf")
     tables = camelot.read_pdf(filename)
-    return camelot.plot(tables[0], kind='text')
+    return camelot.plot(tables[0], kind="text")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_grid_plot():
     filename = os.path.join(testdir, "foo.pdf")
     tables = camelot.read_pdf(filename)
-    return camelot.plot(tables[0], kind='grid')
+    return camelot.plot(tables[0], kind="grid")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_lattice_contour_plot():
     filename = os.path.join(testdir, "foo.pdf")
     tables = camelot.read_pdf(filename)
-    return camelot.plot(tables[0], kind='contour')
+    return camelot.plot(tables[0], kind="contour")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_stream_contour_plot():
     filename = os.path.join(testdir, "tabula/12s0324.pdf")
-    tables = camelot.read_pdf(filename, flavor='stream')
-    return camelot.plot(tables[0], kind='contour')
+    tables = camelot.read_pdf(filename, flavor="stream")
+    return camelot.plot(tables[0], kind="contour")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_line_plot():
     filename = os.path.join(testdir, "foo.pdf")
     tables = camelot.read_pdf(filename)
-    return camelot.plot(tables[0], kind='line')
+    return camelot.plot(tables[0], kind="line")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_joint_plot():
     filename = os.path.join(testdir, "foo.pdf")
     tables = camelot.read_pdf(filename)
-    return camelot.plot(tables[0], kind='joint')
+    return camelot.plot(tables[0], kind="joint")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_textedge_plot():
     filename = os.path.join(testdir, "tabula/12s0324.pdf")
-    tables = camelot.read_pdf(filename, flavor='stream')
-    return camelot.plot(tables[0], kind='textedge')
+    tables = camelot.read_pdf(filename, flavor="stream")
+    return camelot.plot(tables[0], kind="textedge")