Merge pull request #175 from camelot-dev/revert-0-8-1

[MRG] Revert the changes in v0.8.1

.deepsource.toml

@@ -0,0 +1,12 @@
+version = 1
+
+exclude_patterns = [
+  "camelot/ext/**"
+]
+
+[[analyzers]]
+name = "python"
+enabled = true
+
+[analyzers.meta]
+runtime_version = "3.x.x"

.github/FUNDING.yml

@@ -0,0 +1 @@
+open_collective: camelot

.readthedocs.yml

@@ -0,0 +1,24 @@
+# .readthedocs.yml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: docs/conf.py
+
+# Build documentation with MkDocs
+#mkdocs:
+#  configuration: mkdocs.yml
+
+# Optionally build your docs in additional formats such as PDF
+formats:
+  - pdf
+
+# Optionally set the version of Python and requirements required to build your docs
+python:
+  version: 3.8
+  install:
+    - requirements: requirements.txt

.travis.yml

@@ -8,14 +8,6 @@ install:
   - make install
 jobs:
   include:
-    - stage: test
-      script:
-        - make test
-      python: '2.7'
-    - stage: test
-      script:
-        - make test
-      python: '3.5'
     - stage: test
       script:
         - make test
@@ -25,8 +17,13 @@ jobs:
         - make test
       python: '3.7'
       dist: xenial
+    - stage: test
+      script:
+        - make test
+      python: '3.8'
+      dist: xenial
     - stage: coverage
-      python: '3.6'
+      python: '3.8'
       script:
         - make test
         - codecov --verbose

CONTRIBUTING.md

@@ -23,7 +23,7 @@ A great way to start contributing to Camelot is to pick an issue tagged with the
 To install the dependencies needed for development, you can use pip:
 
 <pre>
-$ pip install camelot-py[dev]
+$ pip install "camelot-py[dev]"
 </pre>
 
 Alternatively, you can clone the project repository, and install using pip:

HISTORY.md

@@ -4,8 +4,48 @@ Release History
 master
 ------
 
+0.8.2 (2020-07-27)
+------------------
+
+* Revert the changes in `0.8.1`.
+
+0.8.1 (2020-07-21)
+------------------
+
 **Bugfixes**
 
+* [#169](https://github.com/camelot-dev/camelot/issues/169) Fix import error caused by `pdfminer.six==20200720`. [#171](https://github.com/camelot-dev/camelot/pull/171) by Vinayak Mehta.
+
+0.8.0 (2020-05-24)
+------------------
+
+**Improvements**
+
+* Drop Python 2 support!
+    * Remove Python 2.7 and 3.5 support.
+    * Replace all instances of `.format` with f-strings.
+    * Remove all `__future__` imports.
+    * Fix HTTP 403 forbidden exception in read_pdf(url) and remove Python 2 urllib support.
+    * Fix test data.
+
+**Bugfixes**
+
+* Fix library discovery on Windows. [#32](https://github.com/camelot-dev/camelot/pull/32) by [KOLANICH](https://github.com/KOLANICH).
+* Fix calling convention of callback functions. [#34](https://github.com/camelot-dev/camelot/pull/34) by [KOLANICH](https://github.com/KOLANICH).
+
+0.7.3 (2019-07-07)
+------------------
+
+**Improvements**
+
+* Camelot now follows the Black code style! [#1](https://github.com/camelot-dev/camelot/pull/1) and [#3](https://github.com/camelot-dev/camelot/pull/3).
+
+**Bugfixes**
+
+* Fix Click.HelpFormatter monkey-patch. [#5](https://github.com/camelot-dev/camelot/pull/5) by [Dimiter Naydenov](https://github.com/dimitern).
+* Fix strip_text argument getting ignored. [#4](https://github.com/camelot-dev/camelot/pull/4) by [Dimiter Naydenov](https://github.com/dimitern).
+* [#25](https://github.com/camelot-dev/camelot/issues/25) edge_tol skipped in read_pdf. [#26](https://github.com/camelot-dev/camelot/pull/26) by Vinayak Mehta.
+* Fix pytest deprecation warning. [#2](https://github.com/camelot-dev/camelot/pull/2) by Vinayak Mehta.
 * [#293](https://github.com/socialcopsdev/camelot/issues/293) Split text ignores all text to the right of last cut. [#294](https://github.com/socialcopsdev/camelot/pull/294) by Vinayak Mehta.
 * [#277](https://github.com/socialcopsdev/camelot/issues/277) Sort TableList by order of tables in PDF. [#283](https://github.com/socialcopsdev/camelot/pull/283) by [Sym Roe](https://github.com/symroe).
 * [#312](https://github.com/socialcopsdev/camelot/issues/312) `table_regions` throws `ValueError` when `flavor='stream'`. [#332](https://github.com/socialcopsdev/camelot/pull/332) by Vinayak Mehta.

README.md

@@ -7,7 +7,7 @@
 [![Build Status](https://travis-ci.org/camelot-dev/camelot.svg?branch=master)](https://travis-ci.org/camelot-dev/camelot) [![Documentation Status](https://readthedocs.org/projects/camelot-py/badge/?version=master)](https://camelot-py.readthedocs.io/en/master/)
  [![codecov.io](https://codecov.io/github/camelot-dev/camelot/badge.svg?branch=master&service=github)](https://codecov.io/github/camelot-dev/camelot?branch=master)
  [![image](https://img.shields.io/pypi/v/camelot-py.svg)](https://pypi.org/project/camelot-py/) [![image](https://img.shields.io/pypi/l/camelot-py.svg)](https://pypi.org/project/camelot-py/) [![image](https://img.shields.io/pypi/pyversions/camelot-py.svg)](https://pypi.org/project/camelot-py/) [![Gitter chat](https://badges.gitter.im/camelot-dev/Lobby.png)](https://gitter.im/camelot-dev/Lobby)
-[![image](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black)
+[![image](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/ambv/black) [![image](https://img.shields.io/badge/continous%20quality-deepsource-lightgrey)](https://deepsource.io/gh/camelot-dev/camelot/?ref=repository-badge)
 
 
 **Camelot** is a Python library that makes it easy for *anyone* to extract tables from PDF files!
@@ -71,10 +71,10 @@ $ conda install -c conda-forge camelot-py
 
 ### Using pip
 
-After [installing the dependencies](https://camelot-py.readthedocs.io/en/master/user/install-deps.html) ([tk](https://packages.ubuntu.com/trusty/python-tk) and [ghostscript](https://www.ghostscript.com/)), you can simply use pip to install Camelot:
+After [installing the dependencies](https://camelot-py.readthedocs.io/en/master/user/install-deps.html) ([tk](https://packages.ubuntu.com/bionic/python/python-tk) and [ghostscript](https://www.ghostscript.com/)), you can simply use pip to install Camelot:
 
 <pre>
-$ pip install camelot-py[cv]
+$ pip install "camelot-py[cv]"
 </pre>
 
 ### From the source code
@@ -113,7 +113,7 @@ $ git clone https://www.github.com/camelot-dev/camelot
 You can install the development dependencies easily, using pip:
 
 <pre>
-$ pip install camelot-py[dev]
+$ pip install "camelot-py[dev]"
 </pre>
 
 ### Testing
@@ -124,6 +124,10 @@ After installation, you can run tests using:
 $ python setup.py test
 </pre>
 
+## Wrappers
+
+- [camelot-php](https://github.com/randomstate/camelot-php) provides a [PHP](https://www.php.net/) wrapper on Camelot.
+
 ## Versioning
 
 Camelot uses [Semantic Versioning](https://semver.org/). For the available versions, see the tags on this repository. For the changelog, you can check out [HISTORY.md](https://github.com/camelot-dev/camelot/blob/master/HISTORY.md).
@@ -132,4 +136,8 @@ Camelot uses [Semantic Versioning](https://semver.org/). For the available versi
 
 This project is licensed under the MIT License, see the [LICENSE](https://github.com/camelot-dev/camelot/blob/master/LICENSE) file for details.
 
-<img src="http://i65.tinypic.com/9h4ajs.png" align="centre" />
+## Support the development
+
+You can support our work on Camelot with a one-time or monthly donation [on OpenCollective](https://opencollective.com/camelot). Organizations who use camelot can also sponsor the project for an acknowledgement on [our documentation site](https://camelot-py.readthedocs.io/en/master/) and this README.
+
+Special thanks to all the users, organizations and contributors that support Camelot!

camelot/__main__.py

@@ -1,7 +1,5 @@
 # -*- coding: utf-8 -*-
 
-from __future__ import absolute_import
-
 
 __all__ = ("main",)
 

camelot/__version__.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
 
-VERSION = (0, 7, 2)
+VERSION = (0, 8, 2)
 PRERELEASE = None  # alpha, beta or rc
 REVISION = None
 
@@ -8,9 +8,9 @@ REVISION = None
 def generate_version(version, prerelease=None, revision=None):
     version_parts = [".".join(map(str, version))]
     if prerelease is not None:
-        version_parts.append("-{}".format(prerelease))
+        version_parts.append(f"-{prerelease}")
     if revision is not None:
-        version_parts.append(".{}".format(revision))
+        version_parts.append(f".{revision}")
     return "".join(version_parts)
 
 

camelot/cli.py

@@ -204,7 +204,7 @@ def lattice(c, *args, **kwargs):
     tables = read_pdf(
         filepath, pages=pages, flavor="lattice", suppress_stdout=quiet, **kwargs
     )
-    click.echo("Found {} tables".format(tables.n))
+    click.echo(f"Found {tables.n} tables")
     if plot_type is not None:
         for table in tables:
             plot(table, kind=plot_type)
@@ -295,7 +295,7 @@ def stream(c, *args, **kwargs):
     tables = read_pdf(
         filepath, pages=pages, flavor="stream", suppress_stdout=quiet, **kwargs
     )
-    click.echo("Found {} tables".format(tables.n))
+    click.echo(f"Found {tables.n} tables")
     if plot_type is not None:
         for table in tables:
             plot(table, kind=plot_type)

camelot/core.py

@@ -52,13 +52,10 @@ class TextEdge(object):
         self.is_valid = False
 
     def __repr__(self):
-        return "<TextEdge x={} y0={} y1={} align={} valid={}>".format(
-            round(self.x, 2),
-            round(self.y0, 2),
-            round(self.y1, 2),
-            self.align,
-            self.is_valid,
-        )
+        x = round(self.x, 2)
+        y0 = round(self.y0, 2)
+        y1 = round(self.y1, 2)
+        return f"<TextEdge x={x} y0={y0} y1={y1} align={self.align} valid={self.is_valid}>"
 
     def update_coords(self, x, y0, edge_tol=50):
         """Updates the text edge's x and bottom y coordinates and sets
@@ -291,9 +288,11 @@ class Cell(object):
         self._text = ""
 
     def __repr__(self):
-        return "<Cell x1={} y1={} x2={} y2={}>".format(
-            round(self.x1, 2), round(self.y1, 2), round(self.x2, 2), round(self.y2, 2)
-        )
+        x1 = round(self.x1, 2)
+        y1 = round(self.y1, 2)
+        x2 = round(self.x2, 2)
+        y2 = round(self.y2, 2)
+        return f"<Cell x1={x1} y1={y1} x2={x2} y2={y2}>"
 
     @property
     def text(self):
@@ -351,7 +350,7 @@ class Table(object):
         self.page = None
 
     def __repr__(self):
-        return "<{} shape={}>".format(self.__class__.__name__, self.shape)
+        return f"<{self.__class__.__name__} shape={self.shape}>"
 
     def __lt__(self, other):
         if self.page == other.page:
@@ -612,7 +611,7 @@ class Table(object):
 
         """
         kw = {
-            "sheet_name": "page-{}-table-{}".format(self.page, self.order),
+            "sheet_name": f"page-{self.page}-table-{self.order}",
             "encoding": "utf-8",
         }
         kw.update(kwargs)
@@ -649,7 +648,7 @@ class Table(object):
         kw = {"if_exists": "replace", "index": False}
         kw.update(kwargs)
         conn = sqlite3.connect(path)
-        table_name = "page-{}-table-{}".format(self.page, self.order)
+        table_name = f"page-{self.page}-table-{self.order}"
         self.df.to_sql(table_name, conn, **kw)
         conn.commit()
         conn.close()
@@ -670,7 +669,7 @@ class TableList(object):
         self._tables = tables
 
     def __repr__(self):
-        return "<{} n={}>".format(self.__class__.__name__, self.n)
+        return f"<{self.__class__.__name__} n={self.n}>"
 
     def __len__(self):
         return len(self._tables)
@@ -680,7 +679,7 @@ class TableList(object):
 
     @staticmethod
     def _format_func(table, f):
-        return getattr(table, "to_{}".format(f))
+        return getattr(table, f"to_{f}")
 
     @property
     def n(self):
@@ -691,9 +690,7 @@ class TableList(object):
         root = kwargs.get("root")
         ext = kwargs.get("ext")
         for table in self._tables:
-            filename = os.path.join(
-                "{}-page-{}-table-{}{}".format(root, table.page, table.order, ext)
-            )
+            filename = f"{root}-page-{table.page}-table-{table.order}{ext}"
             filepath = os.path.join(dirname, filename)
             to_format = self._format_func(table, f)
             to_format(filepath)
@@ -706,9 +703,7 @@ class TableList(object):
         zipname = os.path.join(os.path.dirname(path), root) + ".zip"
         with zipfile.ZipFile(zipname, "w", allowZip64=True) as z:
             for table in self._tables:
-                filename = os.path.join(
-                    "{}-page-{}-table-{}{}".format(root, table.page, table.order, ext)
-                )
+                filename = f"{root}-page-{table.page}-table-{table.order}{ext}"
                 filepath = os.path.join(dirname, filename)
                 z.write(filepath, os.path.basename(filepath))
 
@@ -741,7 +736,7 @@ class TableList(object):
             filepath = os.path.join(dirname, basename)
             writer = pd.ExcelWriter(filepath)
             for table in self._tables:
-                sheet_name = "page-{}-table-{}".format(table.page, table.order)
+                sheet_name = f"page-{table.page}-table-{table.order}"
                 table.df.to_excel(writer, sheet_name=sheet_name, encoding="utf-8")
             writer.save()
             if compress:

camelot/ext/ghostscript/_gsprint.py

@@ -82,7 +82,10 @@ def delete_instance(instance):
     return libgs.gsapi_delete_instance(instance)
 
 
-c_stdstream_call_t = CFUNCTYPE(c_int, gs_main_instance, POINTER(c_char), c_int)
+if sys.platform == "win32":
+    c_stdstream_call_t = WINFUNCTYPE(c_int, gs_main_instance, POINTER(c_char), c_int)
+else:
+    c_stdstream_call_t = CFUNCTYPE(c_int, gs_main_instance, POINTER(c_char), c_int)
 
 
 def _wrap_stdin(infp):
@@ -243,6 +246,12 @@ def __win32_finddll():
 
 if sys.platform == "win32":
     libgs = __win32_finddll()
+    if not libgs:
+        import ctypes.util
+
+        libgs = ctypes.util.find_library(
+            "".join(("gsdll", str(ctypes.sizeof(ctypes.c_voidp) * 8), ".dll"))
+        )  # finds in %PATH%
     if not libgs:
         raise RuntimeError("Please make sure that Ghostscript is installed")
     libgs = windll.LoadLibrary(libgs)

camelot/handlers.py

@@ -106,7 +106,7 @@ class PDFHandler(object):
             infile = PdfFileReader(fileobj, strict=False)
             if infile.isEncrypted:
                 infile.decrypt(self.password)
-            fpath = os.path.join(temp, "page-{0}.pdf".format(page))
+            fpath = os.path.join(temp, f"page-{page}.pdf")
             froot, fext = os.path.splitext(fpath)
             p = infile.getPage(page - 1)
             outfile = PdfFileWriter()
@@ -164,7 +164,7 @@ class PDFHandler(object):
             for p in self.pages:
                 self._save_page(self.filepath, p, tempdir)
             pages = [
-                os.path.join(tempdir, "page-{0}.pdf".format(p)) for p in self.pages
+                os.path.join(tempdir, f"page-{p}.pdf") for p in self.pages
             ]
             parser = Lattice(**kwargs) if flavor == "lattice" else Stream(**kwargs)
             for p in pages:

camelot/image_processing.py

@@ -1,7 +1,5 @@
 # -*- coding: utf-8 -*-
 
-from __future__ import division
-
 import cv2
 import numpy as np
 

camelot/parsers/lattice.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 
-from __future__ import division
 import os
 import sys
 import copy

camelot/parsers/stream.py

@@ -1,6 +1,5 @@
 # -*- coding: utf-8 -*-
 
-from __future__ import division
 import os
 import logging
 import warnings
@@ -358,7 +357,7 @@ class Stream(BaseParser):
                     ncols = max(set(elements), key=elements.count)
                 else:
                     warnings.warn(
-                        "No tables found in table area {}".format(table_idx + 1)
+                        f"No tables found in table area {table_idx + 1}"
                     )
             cols = [(t.x0, t.x1) for r in rows_grouped if len(r) == ncols for t in r]
             cols = self._merge_columns(sorted(cols), column_tol=self.column_tol)
@@ -433,19 +432,19 @@ class Stream(BaseParser):
 
     def extract_tables(self, filename, suppress_stdout=False, layout_kwargs={}):
         self._generate_layout(filename, layout_kwargs)
+        base_filename = os.path.basename(self.rootname)
+
         if not suppress_stdout:
-            logger.info("Processing {}".format(os.path.basename(self.rootname)))
+            logger.info(f"Processing {base_filename}")
 
         if not self.horizontal_text:
             if self.images:
                 warnings.warn(
-                    "{} is image-based, camelot only works on"
-                    " text-based pages.".format(os.path.basename(self.rootname))
+                    f"{base_filename} is image-based, camelot only works on"
+                    " text-based pages."
                 )
             else:
-                warnings.warn(
-                    "No tables found on {}".format(os.path.basename(self.rootname))
-                )
+                warnings.warn(f"No tables found on {base_filename}")
             return []
 
         self._generate_table_bbox()

camelot/plotting.py

@@ -35,11 +35,11 @@ class PlotMethods(object):
 
         if table.flavor == "lattice" and kind in ["textedge"]:
             raise NotImplementedError(
-                "Lattice flavor does not support kind='{}'".format(kind)
+                f"Lattice flavor does not support kind='{kind}'"
             )
         elif table.flavor == "stream" and kind in ["joint", "line"]:
             raise NotImplementedError(
-                "Stream flavor does not support kind='{}'".format(kind)
+                f"Stream flavor does not support kind='{kind}'"
             )
 
         plot_method = getattr(self, kind)

camelot/utils.py

@@ -1,9 +1,7 @@
 # -*- coding: utf-8 -*-
-from __future__ import division
 
-import re
 import os
-import sys
+import re
 import random
 import shutil
 import string
@@ -29,16 +27,9 @@ from pdfminer.layout import (
     LTImage,
 )
 
-
-PY3 = sys.version_info[0] >= 3
-if PY3:
-    from urllib.request import urlopen
-    from urllib.parse import urlparse as parse_url
-    from urllib.parse import uses_relative, uses_netloc, uses_params
-else:
-    from urllib2 import urlopen
-    from urlparse import urlparse as parse_url
-    from urlparse import uses_relative, uses_netloc, uses_params
+from urllib.request import Request, urlopen
+from urllib.parse import urlparse as parse_url
+from urllib.parse import uses_relative, uses_netloc, uses_params
 
 
 _VALID_URLS = set(uses_relative + uses_netloc + uses_params)
@@ -88,13 +79,12 @@ def download_url(url):
         Temporary filepath.
 
     """
-    filename = "{}.pdf".format(random_string(6))
+    filename = f"{random_string(6)}.pdf"
     with tempfile.NamedTemporaryFile("wb", delete=False) as f:
-        obj = urlopen(url)
-        if PY3:
+        headers = {"User-Agent": "Mozilla/5.0"}
+        request = Request(url, None, headers)
+        obj = urlopen(request)
         content_type = obj.info().get_content_type()
-        else:
-            content_type = obj.info().getheader("Content-Type")
         if content_type != "application/pdf":
             raise NotImplementedError("File format not supported")
         f.write(obj.read())
@@ -123,9 +113,7 @@ def validate_input(kwargs, flavor="lattice"):
         isec = set(parser_kwargs).intersection(set(input_kwargs.keys()))
         if isec:
             raise ValueError(
-                "{} cannot be used with flavor='{}'".format(
-                    ",".join(sorted(isec)), flavor
-                )
+                f"{','.join(sorted(isec))} cannot be used with flavor='{flavor}'"
             )
 
     if flavor == "lattice":
@@ -423,7 +411,7 @@ def text_strip(text, strip=""):
         return text
 
     stripped = re.sub(
-        r"[{}]".format("".join(map(re.escape, strip))), "", text, re.UNICODE
+        fr"[{''.join(map(re.escape, strip))}]", "", text, re.UNICODE
     )
     return stripped
 
@@ -660,9 +648,7 @@ def get_table_index(
                 text_range = (t.x0, t.x1)
                 col_range = (table.cols[0][0], table.cols[-1][1])
                 warnings.warn(
-                    "{} {} does not lie in column range {}".format(
-                        text, text_range, col_range
-                    )
+                    f"{text} {text_range} does not lie in column range {col_range}"
                 )
             r_idx = r
             c_idx = lt_col_overlap.index(max(lt_col_overlap))
@@ -794,7 +780,7 @@ def get_page_layout(
         parser = PDFParser(f)
         document = PDFDocument(parser)
         if not document.is_extractable:
-            raise PDFTextExtractionNotAllowed
+            raise PDFTextExtractionNotAllowed(f"Text extraction is not allowed: {filename}")
         laparams = LAParams(
             char_margin=char_margin,
             line_margin=line_margin,

docs/dev/contributing.rst

@@ -37,7 +37,7 @@ Setting up a development environment
 
 To install the dependencies needed for development, you can use pip::
 
-    $ pip install camelot-py[dev]
+    $ pip install "camelot-py[dev]"
 
 Alternatively, you can clone the project repository, and install using pip::
 

docs/index.rst

@@ -30,6 +30,12 @@ Release v\ |version|. (:ref:`Installation <install>`)
 .. image:: https://badges.gitter.im/camelot-dev/Lobby.png
     :target: https://gitter.im/camelot-dev/Lobby
 
+.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
+    :target: https://github.com/ambv/black
+
+.. image:: https://img.shields.io/badge/continous%20quality-deepsource-lightgrey
+    :target: https://deepsource.io/gh/camelot-dev/camelot/?ref=repository-badge
+
 **Camelot** is a Python library that makes it easy for *anyone* to extract tables from PDF files!
 
 .. note:: You can also check out `Excalibur`_, which is a web interface for Camelot!
@@ -83,6 +89,13 @@ See `comparison with other PDF table extraction libraries and tools`_.
 .. _ETL and data analysis workflows: https://gist.github.com/vinayak-mehta/e5949f7c2410a0e12f25d3682dc9e873
 .. _comparison with other PDF table extraction libraries and tools: https://github.com/camelot-dev/camelot/wiki/Comparison-with-other-PDF-Table-Extraction-libraries-and-tools
 
+Support us on OpenCollective
+----------------------------
+
+If Camelot helped you extract tables from PDFs, please consider supporting its development by `becoming a backer or a sponsor on OpenCollective`_!
+
+.. _becoming a backer or a sponsor on OpenCollective: https://opencollective.com/camelot
+
 The User Guide
 --------------
 

docs/user/how-it-works.rst

@@ -20,7 +20,7 @@ Stream can be used to parse tables that have whitespaces between cells to simula
 
 3. The number of columns inside each table area are then guessed. This is done by calculating the mode of number of words in each text row. Based on this mode, words in each text row are chosen to calculate a list of column *x* ranges.
 
-4. Words that lie inside/outside the current column *x* ranges are then used to extend extend the current list of columns.
+4. Words that lie inside/outside the current column *x* ranges are then used to extend the current list of columns.
 
 5. Finally, a table is formed using the text rows' *y* ranges and column *x* ranges and words found on the page are assigned to the table's cells based on their *x* and *y* coordinates.
 

docs/user/install.rst

@@ -13,7 +13,7 @@ The easiest way to install Camelot is to install it with `conda`_, which is a pa
 
     $ conda install -c conda-forge camelot-py
 
-.. note:: Camelot is available for Python 2.7, 3.5 and 3.6 on Linux, macOS and Windows. For Windows, you will need to install ghostscript which you can get from their `downloads page`_.
+.. note:: Camelot is available for Python 2.7, 3.5, 3.6 and 3.7 on Linux, macOS and Windows. For Windows, you will need to install ghostscript which you can get from their `downloads page`_.
 
 .. _conda: https://conda.io/docs/
 .. _Anaconda: http://docs.continuum.io/anaconda/
@@ -25,7 +25,7 @@ Using pip
 
 After :ref:`installing the dependencies <install_deps>`, which include `Tkinter`_ and `ghostscript`_, you can simply use pip to install Camelot::
 
-    $ pip install camelot-py[cv]
+    $ pip install "camelot-py[cv]"
 
 .. _Tkinter: https://wiki.python.org/moin/TkInter
 .. _ghostscript: https://www.ghostscript.com

requirements.txt

@@ -1,8 +1,10 @@
+chardet>=3.0.4
 click>=6.7
 matplotlib>=2.2.3
 numpy>=1.13.3
 opencv-python>=3.4.2.17
 openpyxl>=2.5.8
 pandas>=0.23.4
-pdfminer.six>=20170720
+pdfminer.six>=20200726
 PyPDF2>=1.26.0
+Sphinx>=3.1.2

setup.py

@@ -19,7 +19,7 @@ requires = [
     'numpy>=1.13.3',
     'openpyxl>=2.5.8',
     'pandas>=0.23.4',
-    'pdfminer.six>=20170720',
+    'pdfminer.six>=20200726',
     'PyPDF2>=1.26.0'
 ]
 
@@ -33,11 +33,11 @@ plot_requires = [
 
 dev_requires = [
     'codecov>=2.0.15',
-    'pytest>=3.8.0',
-    'pytest-cov>=2.6.0',
-    'pytest-mpl>=0.10',
-    'pytest-runner>=4.2',
-    'Sphinx>=1.7.9'
+    'pytest>=5.4.3',
+    'pytest-cov>=2.10.0',
+    'pytest-mpl>=0.11',
+    'pytest-runner>=5.2',
+    'Sphinx>=3.1.2'
 ]
 
 all_requires = cv_requires + plot_requires
@@ -71,10 +71,9 @@ def setup_package():
                         # Trove classifiers
                         # Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers
                         'License :: OSI Approved :: MIT License',
-                        'Programming Language :: Python :: 2.7',
-                        'Programming Language :: Python :: 3.5',
                         'Programming Language :: Python :: 3.6',
-                        'Programming Language :: Python :: 3.7'
+                        'Programming Language :: Python :: 3.7',
+                        'Programming Language :: Python :: 3.8'
                     ])
 
     try:

tests/__init__.py

@@ -1,2 +1,3 @@
 import matplotlib
-matplotlib.use('agg')
+
+matplotlib.use("agg")

tests/data.py

@@ -1,19 +1,7 @@
 # -*- coding: utf-8 -*-
 
-from __future__ import unicode_literals
-
 
 data_stream = [
-    [
-        "",
-        "Table: 5            Public Health Outlay 2012-13 (Budget Estimates)        (Rs. in 000)",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-    ],
     ["States-A", "Revenue", "", "Capital", "", "Total", "Others(1)", "Total"],
     ["", "", "", "", "", "Revenue &", "", ""],
     ["", "Medical &", "Family", "Medical &", "Family", "", "", ""],
@@ -829,18 +817,6 @@ data_stream_table_rotated = [
 ]
 
 data_stream_two_tables_1 = [
-    [
-        "[In thousands (11,062.6 represents 11,062,600) For year ending December 31. Based on Uniform Crime Reporting (UCR)",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-    ],
     [
         "Program. Represents arrests reported (not charged) by 12,910 agencies with a total population of 247,526,916 as estimated",
         "",
@@ -1300,29 +1276,10 @@ data_stream_two_tables_1 = [
         "",
         "",
     ],
-    [
-        "",
-        "Source: U.S. Department of Justice, Federal Bureau of Investigation, Uniform Crime Reports, Arrests Master Files.",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-        "",
-    ],
 ]
 
+
 data_stream_two_tables_2 = [
-    [
-        "",
-        "Source: U.S. Department of Justice, Federal Bureau of Investigation, Uniform Crime Reports, Arrests Master Files.",
-        "",
-        "",
-        "",
-        "",
-    ],
     ["Table 325. Arrests by Race: 2009", "", "", "", "", ""],
     [
         "[Based on Uniform Crime Reporting (UCR) Program. Represents arrests reported (not charged) by 12,371 agencies",
@@ -1600,16 +1557,9 @@ data_stream_two_tables_2 = [
         "3,950",
     ],
     ["1 Except forcible rape and prostitution.", "", "", "", "", ""],
-    [
-        "",
-        "Source: U.S. Department of Justice, Federal Bureau of Investigation, “Crime in the United States, Arrests,” September 2010,",
-        "",
-        "",
-        "",
-        "",
-    ],
 ]
 
+
 data_stream_table_areas = [
     ["", "One Withholding"],
     ["Payroll Period", "Allowance"],
@@ -1776,18 +1726,7 @@ data_stream_columns = [
 ]
 
 data_stream_split_text = [
-    [
-        "FEB",
-        "RUAR",
-        "Y 2014 M27 (BUS)",
-        "",
-        "ALPHABETIC LISTING BY T",
-        "YPE",
-        "",
-        "",
-        "",
-        "ABLPDM27",
-    ],
+    ["FEB", "RUAR", "Y 2014 M27 (BUS)", "", "", "", "", "", "", ""],
     ["", "", "", "", "OF ACTIVE LICENSES", "", "", "", "", "3/19/2014"],
     ["", "", "", "", "OKLAHOMA ABLE COMMIS", "SION", "", "", "", ""],
     ["LICENSE", "", "", "", "PREMISE", "", "", "", "", ""],
@@ -2121,6 +2060,7 @@ data_stream_split_text = [
     ],
 ]
 
+
 data_stream_flag_size = [
     [
         "States",
@@ -2820,7 +2760,7 @@ data_arabic = [
 ]
 
 data_stream_layout_kwargs = [
-    ["V i n s   a u   Ve r r e", ""],
+    ["V i n s   a u   V e r r e", ""],
     ["Les Blancs", "12.5CL"],
     ["A.O.P Côtes du Rhône", ""],
     ["Domaine de la Guicharde «  Autour de la chapelle » 2016", "8 €"],

tests/test_cli.py

@@ -114,31 +114,35 @@ def test_cli_password():
 def test_cli_output_format():
     with TemporaryDirectory() as tempdir:
         infile = os.path.join(testdir, "health.pdf")
-        outfile = os.path.join(tempdir, "health.{}")
+
         runner = CliRunner()
 
         # json
+        outfile = os.path.join(tempdir, "health.json")
         result = runner.invoke(
             cli,
-            ["--format", "json", "--output", outfile.format("json"), "stream", infile],
+            ["--format", "json", "--output", outfile, "stream", infile],
         )
         assert result.exit_code == 0
 
         # excel
+        outfile = os.path.join(tempdir, "health.xlsx")
         result = runner.invoke(
             cli,
-            ["--format", "excel", "--output", outfile.format("xlsx"), "stream", infile],
+            ["--format", "excel", "--output", outfile, "stream", infile],
         )
         assert result.exit_code == 0
 
         # html
+        outfile = os.path.join(tempdir, "health.html")
         result = runner.invoke(
             cli,
-            ["--format", "html", "--output", outfile.format("html"), "stream", infile],
+            ["--format", "html", "--output", outfile, "stream", infile],
         )
         assert result.exit_code == 0
 
         # zip
+        outfile = os.path.join(tempdir, "health.csv")
         result = runner.invoke(
             cli,
             [
@@ -146,7 +150,7 @@ def test_cli_output_format():
                 "--format",
                 "csv",
                 "--output",
-                outfile.format("csv"),
+                outfile,
                 "stream",
                 infile,
             ],

tests/test_common.py

@@ -3,9 +3,11 @@
 import os
 
 import pandas as pd
+from pandas.testing import assert_frame_equal
 
 import camelot
 from camelot.core import Table, TableList
+from camelot.__version__ import generate_version
 
 from .data import *
 
@@ -26,10 +28,10 @@ def test_password():
 
     filename = os.path.join(testdir, "health_protected.pdf")
     tables = camelot.read_pdf(filename, password="ownerpass", flavor="stream")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
     tables = camelot.read_pdf(filename, password="userpass", flavor="stream")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream():
@@ -37,7 +39,7 @@ def test_stream():
 
     filename = os.path.join(testdir, "health.pdf")
     tables = camelot.read_pdf(filename, flavor="stream")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_table_rotated():
@@ -45,11 +47,11 @@ def test_stream_table_rotated():
 
     filename = os.path.join(testdir, "clockwise_table_2.pdf")
     tables = camelot.read_pdf(filename, flavor="stream")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
     filename = os.path.join(testdir, "anticlockwise_table_2.pdf")
     tables = camelot.read_pdf(filename, flavor="stream")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_two_tables():
@@ -71,7 +73,7 @@ def test_stream_table_regions():
     tables = camelot.read_pdf(
         filename, flavor="stream", table_regions=["320,460,573,335"]
     )
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_table_areas():
@@ -81,7 +83,7 @@ def test_stream_table_areas():
     tables = camelot.read_pdf(
         filename, flavor="stream", table_areas=["320,500,573,335"]
     )
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_columns():
@@ -91,7 +93,7 @@ def test_stream_columns():
     tables = camelot.read_pdf(
         filename, flavor="stream", columns=["67,180,230,425,475"], row_tol=10
     )
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_split_text():
@@ -104,7 +106,7 @@ def test_stream_split_text():
         columns=["72,95,209,327,442,529,566,606,683"],
         split_text=True,
     )
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_flag_size():
@@ -112,7 +114,7 @@ def test_stream_flag_size():
 
     filename = os.path.join(testdir, "superscript.pdf")
     tables = camelot.read_pdf(filename, flavor="stream", flag_size=True)
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_strip_text():
@@ -120,7 +122,7 @@ def test_stream_strip_text():
 
     filename = os.path.join(testdir, "detect_vertical_false.pdf")
     tables = camelot.read_pdf(filename, flavor="stream", strip_text=" ,\n")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_edge_tol():
@@ -128,7 +130,7 @@ def test_stream_edge_tol():
 
     filename = os.path.join(testdir, "edge_tol.pdf")
     tables = camelot.read_pdf(filename, flavor="stream", edge_tol=500)
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_stream_layout_kwargs():
@@ -138,7 +140,7 @@ def test_stream_layout_kwargs():
     tables = camelot.read_pdf(
         filename, flavor="stream", layout_kwargs={"detect_vertical": False}
     )
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_lattice():
@@ -148,7 +150,7 @@ def test_lattice():
         testdir, "tabula/icdar2013-dataset/competition-dataset-us/us-030.pdf"
     )
     tables = camelot.read_pdf(filename, pages="2")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_lattice_table_rotated():
@@ -156,11 +158,11 @@ def test_lattice_table_rotated():
 
     filename = os.path.join(testdir, "clockwise_table_1.pdf")
     tables = camelot.read_pdf(filename)
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
     filename = os.path.join(testdir, "anticlockwise_table_1.pdf")
     tables = camelot.read_pdf(filename)
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_lattice_two_tables():
@@ -179,7 +181,7 @@ def test_lattice_table_regions():
 
     filename = os.path.join(testdir, "table_region.pdf")
     tables = camelot.read_pdf(filename, table_regions=["170,370,560,270"])
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_lattice_table_areas():
@@ -187,7 +189,7 @@ def test_lattice_table_areas():
 
     filename = os.path.join(testdir, "twotables_2.pdf")
     tables = camelot.read_pdf(filename, table_areas=["80,693,535,448"])
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_lattice_process_background():
@@ -195,7 +197,7 @@ def test_lattice_process_background():
 
     filename = os.path.join(testdir, "background_lines_1.pdf")
     tables = camelot.read_pdf(filename, process_background=True)
-    assert df.equals(tables[1].df)
+    assert_frame_equal(df, tables[1].df)
 
 
 def test_lattice_copy_text():
@@ -203,7 +205,7 @@ def test_lattice_copy_text():
 
     filename = os.path.join(testdir, "row_span_1.pdf")
     tables = camelot.read_pdf(filename, line_scale=60, copy_text="v")
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_lattice_shift_text():
@@ -271,7 +273,7 @@ def test_arabic():
 
     filename = os.path.join(testdir, "tabula/arabic.pdf")
     tables = camelot.read_pdf(filename)
-    assert df.equals(tables[0].df)
+    assert_frame_equal(df, tables[0].df)
 
 
 def test_table_order():
@@ -297,3 +299,18 @@ def test_table_order():
         (1, 2),
         (1, 1),
     ]
+
+
+def test_version_generation():
+    version = (0, 7, 3)
+    assert generate_version(version, prerelease=None, revision=None) == "0.7.3"
+
+
+def test_version_generation_with_prerelease_revision():
+    version = (0, 7, 3)
+    prerelease = "alpha"
+    revision = 2
+    assert (
+        generate_version(version, prerelease=prerelease, revision=revision)
+        == "0.7.3-alpha.2"
+    )

tests/test_errors.py

@@ -10,88 +10,93 @@ import camelot
 
 testdir = os.path.dirname(os.path.abspath(__file__))
 testdir = os.path.join(testdir, "files")
-filename = os.path.join(testdir, 'foo.pdf')
+filename = os.path.join(testdir, "foo.pdf")
 
 
 def test_unknown_flavor():
-    message = ("Unknown flavor specified."
-               " Use either 'lattice' or 'stream'")
+    message = "Unknown flavor specified." " Use either 'lattice' or 'stream'"
     with pytest.raises(NotImplementedError, match=message):
-        tables = camelot.read_pdf(filename, flavor='chocolate')
+        tables = camelot.read_pdf(filename, flavor="chocolate")
 
 
 def test_input_kwargs():
     message = "columns cannot be used with flavor='lattice'"
     with pytest.raises(ValueError, match=message):
-        tables = camelot.read_pdf(filename, columns=['10,20,30,40'])
+        tables = camelot.read_pdf(filename, columns=["10,20,30,40"])
 
 
 def test_unsupported_format():
-    message = 'File format not supported'
-    filename = os.path.join(testdir, 'foo.csv')
+    message = "File format not supported"
+    filename = os.path.join(testdir, "foo.csv")
     with pytest.raises(NotImplementedError, match=message):
         tables = camelot.read_pdf(filename)
 
 
 def test_stream_equal_length():
-    message = ("Length of table_areas and columns"
-               " should be equal")
+    message = "Length of table_areas and columns" " should be equal"
     with pytest.raises(ValueError, match=message):
-        tables = camelot.read_pdf(filename, flavor='stream',
-            table_areas=['10,20,30,40'], columns=['10,20,30,40', '10,20,30,40'])
+        tables = camelot.read_pdf(
+            filename,
+            flavor="stream",
+            table_areas=["10,20,30,40"],
+            columns=["10,20,30,40", "10,20,30,40"],
+        )
 
 
 def test_image_warning():
-    filename = os.path.join(testdir, 'image.pdf')
+    filename = os.path.join(testdir, "image.pdf")
     with warnings.catch_warnings():
-        warnings.simplefilter('error')
+        warnings.simplefilter("error")
         with pytest.raises(UserWarning) as e:
             tables = camelot.read_pdf(filename)
-        assert str(e.value) == 'page-1 is image-based, camelot only works on text-based pages.'
+            assert (
+                str(e.value)
+                == "page-1 is image-based, camelot only works on text-based pages."
+            )
 
 
 def test_no_tables_found():
-    filename = os.path.join(testdir, 'blank.pdf')
+    filename = os.path.join(testdir, "blank.pdf")
     with warnings.catch_warnings():
-        warnings.simplefilter('error')
+        warnings.simplefilter("error")
         with pytest.raises(UserWarning) as e:
             tables = camelot.read_pdf(filename)
-        assert str(e.value) == 'No tables found on page-1'
+        assert str(e.value) == "No tables found on page-1"
 
 
 def test_no_tables_found_logs_suppressed():
-    filename = os.path.join(testdir, 'foo.pdf')
+    filename = os.path.join(testdir, "foo.pdf")
     with warnings.catch_warnings():
         # the test should fail if any warning is thrown
-        warnings.simplefilter('error')
+        warnings.simplefilter("error")
         try:
             tables = camelot.read_pdf(filename, suppress_stdout=True)
         except Warning as e:
             warning_text = str(e)
-            pytest.fail('Unexpected warning: {}'.format(warning_text))
+            pytest.fail(f"Unexpected warning: {warning_text}")
 
 
 def test_no_tables_found_warnings_suppressed():
-    filename = os.path.join(testdir, 'blank.pdf')
+    filename = os.path.join(testdir, "blank.pdf")
     with warnings.catch_warnings():
         # the test should fail if any warning is thrown
-        warnings.simplefilter('error')
+        warnings.simplefilter("error")
         try:
             tables = camelot.read_pdf(filename, suppress_stdout=True)
         except Warning as e:
             warning_text = str(e)
-            pytest.fail('Unexpected warning: {}'.format(warning_text))
+            pytest.fail(f"Unexpected warning: {warning_text}")
 
 
 def test_no_password():
-    filename = os.path.join(testdir, 'health_protected.pdf')
-    message = 'file has not been decrypted'
+    filename = os.path.join(testdir, "health_protected.pdf")
+    message = "file has not been decrypted"
     with pytest.raises(Exception, match=message):
         tables = camelot.read_pdf(filename)
 
 
 def test_bad_password():
-    filename = os.path.join(testdir, 'health_protected.pdf')
-    message = 'file has not been decrypted'
+    filename = os.path.join(testdir, "health_protected.pdf")
+    message = "file has not been decrypted"
     with pytest.raises(Exception, match=message):
-        tables = camelot.read_pdf(filename, password='wrongpass')
+        tables = camelot.read_pdf(filename, password="wrongpass")

tests/test_plotting.py

@@ -11,57 +11,50 @@ testdir = os.path.dirname(os.path.abspath(__file__))
 testdir = os.path.join(testdir, "files")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_text_plot():
     filename = os.path.join(testdir, "foo.pdf")
     tables = camelot.read_pdf(filename)
-    return camelot.plot(tables[0], kind='text')
+    return camelot.plot(tables[0], kind="text")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_grid_plot():
     filename = os.path.join(testdir, "foo.pdf")
     tables = camelot.read_pdf(filename)
-    return camelot.plot(tables[0], kind='grid')
+    return camelot.plot(tables[0], kind="grid")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_lattice_contour_plot():
     filename = os.path.join(testdir, "foo.pdf")
     tables = camelot.read_pdf(filename)
-    return camelot.plot(tables[0], kind='contour')
+    return camelot.plot(tables[0], kind="contour")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_stream_contour_plot():
     filename = os.path.join(testdir, "tabula/12s0324.pdf")
-    tables = camelot.read_pdf(filename, flavor='stream')
-    return camelot.plot(tables[0], kind='contour')
+    tables = camelot.read_pdf(filename, flavor="stream")
+    return camelot.plot(tables[0], kind="contour")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_line_plot():
     filename = os.path.join(testdir, "foo.pdf")
     tables = camelot.read_pdf(filename)
-    return camelot.plot(tables[0], kind='line')
+    return camelot.plot(tables[0], kind="line")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_joint_plot():
     filename = os.path.join(testdir, "foo.pdf")
     tables = camelot.read_pdf(filename)
-    return camelot.plot(tables[0], kind='joint')
+    return camelot.plot(tables[0], kind="joint")
 
 
-@pytest.mark.mpl_image_compare(
-    baseline_dir="files/baseline_plots", remove_text=True)
+@pytest.mark.mpl_image_compare(baseline_dir="files/baseline_plots", remove_text=True)
 def test_textedge_plot():
     filename = os.path.join(testdir, "tabula/12s0324.pdf")
-    tables = camelot.read_pdf(filename, flavor='stream')
-    return camelot.plot(tables[0], kind='textedge')
+    tables = camelot.read_pdf(filename, flavor="stream")
+    return camelot.plot(tables[0], kind="textedge")