Feat/tika4

.github/workflows/ci.yml

@@ -36,6 +36,13 @@ jobs:
         python-version: ${{ matrix.python-version }}
         cache: pip
 
+    # Apache Tika 4 requires Java 17+; test against the minimum
+    - name: Set up Java 17
+      uses: actions/setup-java@v6
+      with:
+        distribution: temurin
+        java-version: "17"
+
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip

README.md

@@ -11,14 +11,21 @@ tika-python
 
 A Python port of the [Apache Tika](http://tika.apache.org/)
 library that makes Tika available using the
-[Tika REST Server](https://cwiki.apache.org/confluence/display/TIKA/TikaServer).
+[Tika REST Server](https://tika.apache.org/docs/4.1.x/using-tika/server/index.html).
 
 This makes Apache Tika available as a Python library,
 installable via Setuptools, Pip and Easy Install.
 
-To use this library, you need to have Java 11+ installed on your
+To use this library, you need to have Java 17+ installed on your
 system as tika-python starts up the Tika REST server in the
-background.
+background. Apache Tika 4 runs its parsers in forked worker JVMs that
+are started with the `java` found on the `PATH` (when `TIKA_JAVA`
+points at a specific binary, its directory is put on the server's
+`PATH` for you).
+
+tika-python 4.x targets Apache Tika 4.x and does not work against a
+Tika 3 server; stay on tika-python 3.x for that. See
+"Upgrading from Tika 3.x" below for what changed.
 
 Inspired by [Aptivate Tika](https://github.com/aptivate/python-tika).
 
@@ -28,30 +35,83 @@ Installation (with pip)
 
 Airgap Environment Setup
 ------------------------
-To get this working in a disconnected environment, download a tika server file (both tika-server.jar and tika-server.jar.md5, which can be found [here](https://repo1.maven.org/maven2/org/apache/tika/tika-server-standard/)) and set the TIKA_SERVER_JAR environment variable to TIKA_SERVER_JAR="file:///<yourpath>/tika-server-standard.jar" which successfully tells `python-tika` to "download" this file and move it to `/tmp/tika-server-standard.jar` and run as background process.
+To get this working in a disconnected environment, download the Tika server
+distribution `tika-server-standard-<version>.zip` (and, optionally, its
+`.sha512` checksum file) from the
+[Apache archive](https://archive.apache.org/dist/tika/) and point
+`TIKA_SERVER_JAR` at it:
+
+```bash
+export TIKA_SERVER_JAR="file:///<yourpath>/tika-server-standard-4.1.0.zip"
+# or simply
+export TIKA_SERVER_JAR="/<yourpath>/tika-server-standard-4.1.0.zip"
+```
+
+`python-tika` verifies the zip against the `.sha512` file if it sits next to it
+(otherwise it logs a warning and skips the check), unpacks it once into
+`TIKA_PATH` and starts the server from there. You can also unzip the
+distribution yourself and point `TIKA_SERVER_JAR` at the
+`tika-server-standard-<version>.jar` inside the unzipped directory; that jar
+only runs next to its `lib/` directory.
 
-This is the only way to run `python-tika` without internet access. Without this set, the default is to check the tika version and pull latest every time from Apache.
+This is the only way to run `python-tika` without internet access. Without this
+set, the default is to download the release matching `TIKA_VERSION` from
+`dlcdn.apache.org`, falling back to `archive.apache.org` for releases that are
+no longer on the mirrors.
 
 Environment Variables
 ---------------------
 These are read once, when tika/tika.py is initially loaded and used throughout after that.
 
-1. `TIKA_VERSION` - set to the version string, e.g., 1.12 or default to current Tika version.
-2. `TIKA_SERVER_JAR` - set to the full URL to the remote Tika server jar to download and cache.
-3. `TIKA_SERVER_ENDPOINT` - set to the host (local or remote) for the running Tika server jar.
+1. `TIKA_VERSION` - set to the version string, e.g., 4.1.0 or default to current Tika version.
+2. `TIKA_SERVER_JAR` - URL, `file://` URL or local path of the `tika-server-standard-<version>.zip` distribution to download and cache. May also point at a `tika-server-standard-<version>.jar` that you already unzipped next to its `lib/` directory.
+3. `TIKA_SERVER_ENDPOINT` - set to the host (local or remote) for the running Tika server.
 4. `TIKA_CLIENT_ONLY` - if set to True, then `TIKA_SERVER_JAR` is ignored, and relies on the value for `TIKA_SERVER_ENDPOINT` and treats Tika like a REST client.
-3. `TIKA_JAR_HASH_ALGO` - set to `sha1` when running on FIPS-compliant systems; default value is `md5`.
-4. `TIKA_SERVER_ENDPOINT` - set to the host (local or remote) for the running Tika server jar.
-5. `TIKA_CLIENT_ONLY` - if set to True, then `TIKA_SERVER_JAR` is ignored, and relies on the value for `TIKA_SERVER_ENDPOINT` and treats Tika like a REST client.
-6. `TIKA_TRANSLATOR` - set to the fully qualified class name (defaults to Lingo24) for the Tika translator implementation.
-7. `TIKA_SERVER_CLASSPATH` - set to a string (delimited by ':' for each additional path) to prepend to the Tika server jar path.
-8. `TIKA_LOG_PATH` - set to a directory with write permissions and the `tika.log` and `tika-server.log` files will be placed in this directory.
-9. `TIKA_PATH` - set to a directory with write permissions and the `tika_server.jar` file will be placed in this directory.
-10. `TIKA_JAVA` - set the Java runtime name, e.g., `java` or `java9`
-11. `TIKA_STARTUP_SLEEP` - number of seconds (`float`) to wait per check if Tika server is launched at runtime
-12. `TIKA_STARTUP_MAX_RETRY` - number of checks (`int`) to attempt for Tika server startup if launched at runtime
-13. `TIKA_JAVA_ARGS` - set java runtime arguments, e.g, `-Xmx4g`
-14. `TIKA_LOG_FILE` - set the filename for the log file. default: `tika.log`. if it is an empty string (`''`), no log file is created.
+5. `TIKA_JAR_HASH_ALGO` - hash used to verify the downloaded distribution; default `sha512`, which is the only checksum Apache publishes for the zip (and is FIPS compliant).
+6. `TIKA_SERVER_CLASSPATH` - set to a string (delimited by ':' for each additional path) to prepend to the Tika server jar path.
+7. `TIKA_LOG_PATH` - set to a directory with write permissions and the `tika.log` and `tika-server.log` files will be placed in this directory.
+8. `TIKA_PATH` - set to a directory with write permissions; the server zip, its checksum and the unzipped `tika-server-standard-<version>/` directory are placed there (default: the system temp directory).
+9. `TIKA_JAVA` - set the Java runtime name or path, e.g., `java` or `/usr/lib/jvm/java-17/bin/java`; Java 17 or later is required.
+10. `TIKA_STARTUP_SLEEP` - number of seconds (`float`) to wait per check if Tika server is launched at runtime (default 5)
+11. `TIKA_STARTUP_MAX_RETRY` - number of checks (`int`) to attempt for Tika server startup if launched at runtime (default 6)
+12. `TIKA_JAVA_ARGS` - set java runtime arguments for the server JVM, e.g, `-Xmx4g`. Since Tika 4 parsing happens in forked worker JVMs whose options (`forkedJvmArgs`, `javaPath`, `numClients`) are set in the server's JSON config file, not through this variable.
+13. `TIKA_LOG_FILE` - set the filename for the log file. default: `tika.log`. if it is an empty string (`''`), no log file is created.
+
+Upgrading from Tika 3.x
+=======================
+tika-python 4.x moved from Apache Tika 3.3 to Apache Tika 4.1. Tika 4 changed
+its server in ways that are visible to users of this library:
+
+* **Java 17+** is required (Tika 3 ran on Java 11).
+* **Metadata keys were renamed** and the server does not map them back:
+  `X-TIKA:content` is now `tk:content`, `resourceName` is `tk:resource-name`,
+  `X-TIKA:Parsed-By` is `tk:parsed-by`, `X-TIKA:EXCEPTION:*` is `tk:exception:*`,
+  and many format specific keys are kebab-case now (e.g. `pdf:pdf-version`).
+  `Content-Type` and `Content-Length` are unchanged. `parsed["content"]` keeps
+  working; code reading other metadata keys needs updating.
+* **The server distribution is a zip**, not a runnable jar. `TIKA_SERVER_JAR`
+  now points at `tika-server-standard-<version>.zip`, which is verified with its
+  `.sha512` checksum and unzipped into `TIKA_PATH`. Leftover `tika-server.jar*`
+  files from 3.x in your temp directory can be deleted.
+* **The translate interface was removed** (`tika.translate` and the `translate`
+  CLI command); Tika 4 has no `/translate` endpoint.
+* **Language detection** works on extracted text: `language.from_file` parses
+  the document first. Tika 4 also uses a different detector, which is less
+  reliable on very short snippets.
+* **Unpack returns embedded documents** under `attachments`, keyed by the
+  server's entry names (`1.<ext>`, `2.<ext>`, ...) rather than the original
+  file names.
+* `parser.from_file(..., service='text')` uses `/tika/text` (plain `/tika`
+  returns Markdown in Tika 4), and `service='text', xmlContent=True` now really
+  returns XHTML.
+* **`config_path` must be a JSON file** containing a `server` element; Tika 4
+  no longer reads `tika-config.xml`. Per-request `X-Tika-*` headers are
+  silently ignored by Tika 4.
+* New response codes: 429 when the server's worker pool is saturated, 503 when a
+  worker crashed or timed out, 413 for requests over 1 GiB. Error bodies are
+  JSON; `parsed["status"]` carries the code and `metadata`/`content` are `None`.
+* Memory limits for parsing are set in the server's JSON config
+  (`forkedJvmArgs`), no longer through `TIKA_JAVA_ARGS`.
 
 Testing it out
 ==============
@@ -146,14 +206,22 @@ print(parsed["content"])
 Unpack Interface
 ----------------
 The unpack interface handles both metadata and text extraction in a single
-call and internally returns back a tarball of metadata and text entries that
-is internally unpacked, reducing the wire load for extraction.
+call and additionally returns the documents embedded in the container (images
+in a PDF, members of an archive, attachments of an email, ...). The server
+answers with a zip that is unpacked internally.
 
 ```python
 from tika import unpack
 parsed = unpack.from_file('/path/to/file')
+print(parsed["metadata"])     # metadata of the container document
+print(parsed["content"])      # its extracted text
+for name, data in parsed["attachments"].items():
+    print(name, len(data))    # embedded documents, e.g. '1.png' -> bytes
 ```
 
+Attachment names are the entry names chosen by the server (`1.<ext>`, `2.<ext>`, ...);
+the container document itself is not part of `attachments`.
+
 Detect Interface
 ----------------------
 The detect interface provides a IANA MIME type classification for the
@@ -180,21 +248,14 @@ print(config.getDetectors())
 Language Detection Interface
 ---------------------------------
 The language detection interface provides a 2 character language
-code texted based on the text in provided file.
+code based on the text in the provided file. The server's `/language`
+endpoint expects text, so `from_file` first extracts the text of the
+document and then detects its language.
 
 ```python
 from tika import language
 print(language.from_file('/path/to/file'))
-```
-
-Translate Interface
-------------------------
-The translate interface translates the text automatically extracted
-by Tika from the source language to the destination language.
-
-```python
-from tika import translate
-print(translate.from_file('/path/to/spanish', 'es', 'en'))
+print(language.from_buffer('A few sentences of text are enough to tell the language apart.'))
 ```
 
 Using a Buffer
@@ -253,7 +314,7 @@ print parsed["metadata"]
 
 Customizing the Tika Server Request
 ---------------------------
-You may customize the outgoing HTTP request to Tika server by setting `requestOptions` on the `.from_file` and `.from_buffer` methods (Parser, Unpack , Detect, Config, Language, Translate). It should be a dictionary of arguments that will be passed to the request method. The [request method documentation](https://requests.kennethreitz.org/en/master/api/#requests.request) specifies valid arguments. This will override any defaults except for `url` and `params `/`data`.
+You may customize the outgoing HTTP request to Tika server by setting `requestOptions` on the `.from_file` and `.from_buffer` methods (Parser, Unpack, Detect, Config, Language). It should be a dictionary of arguments that will be passed to the request method. The [request method documentation](https://requests.kennethreitz.org/en/master/api/#requests.request) specifies valid arguments. This will override any defaults except for `url` and `params `/`data`.
 
 ```python
 from tika import parser
@@ -268,18 +329,17 @@ directory.
 
 The options and help for the command line tool can be seen by typing
 `tika-python` without any arguments. This will also download a copy of
-the tika-server jar and start it if you haven't done so already.
+the Tika server distribution and start it if you haven't done so already.
 
 ```bash
-tika.py [-v] [-o <outputDir>] [--server <TikaServerEndpoint>] [--install <UrlToTikaServerJar>] [--port <portNumber>] <command> <option> <urlOrPathToFile>
+tika.py [-v] [-e] [-o <outputDir>] [--server <TikaServerEndpoint>] [--install <UrlToTikaServerZip>] [--port <portNumber>] <command> <option> <urlOrPathToFile>
 
 tika.py parse all test.pdf test2.pdf                   (write output JSON metadata files for test1.pdf_meta.json and test2.pdf_meta.json)
 tika.py detect type test.pdf                           (returns mime-type as text/plain)
 tika.py language file french.txt                       (returns language e.g., fr as text/plain)
-tika.py translate fr:en french.txt                     (translates the file french.txt from french to english)
 tika.py config mime-types                              (see what mime-types the Tika Server can handle)
 
-A simple python and command-line client for Tika using the standalone Tika server (JAR file).
+A simple python and command-line client for Tika using the standalone Tika server (zip distribution).
 All commands return results in JSON format by default (except text in text/plain).
 
 To parse docs, use:
@@ -292,7 +352,6 @@ Commands:
   parse  = parse the input file and write a JSON doc file.ext_meta.json containing the extracted metadata, text, or both
   detect type = parse the stream and 'detect' the MIME/media type, return in text/plain
   language file = parse the file stream and identify the language of the text, return its 2 character code in text/plain
-  translate src:dest = parse and extract text and then translate the text from source language to destination language
   config = return a JSON doc describing the configuration of the Tika server (i.e. mime-types it
              can handle, or installed detectors or parsers)
 
@@ -304,7 +363,7 @@ Switches:
   --encode, -e           = encode response in UTF-8
   --csv, -c    = report detect output in comma-delimited format
   --server <TikaServerEndpoint>  = use a remote Tika Server at this endpoint, otherwise use local server
-  --install <UrlToTikaServerJar> = download and exec Tika Server (JAR file), starting server on default port 9998
+  --install <UrlToTikaServerZip> = download, unpack and exec the Tika Server distribution, starting server on default port 9998
 
 Example usage as python client:
 -- from tika import runCommand, parse1

tests/test_language.py

@@ -13,4 +13,9 @@ def test_local_path(test_file_path):
 
 
 def test_local_buffer():
-    assert language.from_buffer("Good evening, David. How are you?") == "en"
+    assert (
+        language.from_buffer(
+            "Good evening, David. How are you? The weather has been lovely this week and the garden is finally in bloom."
+        )
+        == "en"
+    )

tests/test_parser.py

@@ -1,10 +1,60 @@
 # SPDX-License-Identifier: Apache-2.0
 
+import json
 from http import HTTPStatus
 
 from tika import parser
 
 
+# --- response parsing, no server needed -------------------------------------------
+
+
+def test_parse_concatenates_content_and_merges_metadata():
+    body = json.dumps(
+        [
+            {parser._CONTENT_KEY: "outer ", "Content-Type": "application/pdf", "dc:title": "t"},
+            {parser._CONTENT_KEY: "inner", "Content-Type": "image/png"},
+        ]
+    )
+    parsed = parser._parse((200, body))
+    assert parsed["status"] == 200
+    assert parsed["content"] == "outer inner"
+    assert parsed["metadata"]["Content-Type"] == ["application/pdf", "image/png"]
+    assert parsed["metadata"]["dc:title"] == "t"
+    assert parser._CONTENT_KEY not in parsed["metadata"]
+
+
+def test_parse_without_content_yields_none():
+    parsed = parser._parse((200, json.dumps([{"Content-Type": "image/png"}])))
+    assert parsed["content"] is None
+    assert parsed["metadata"] == {"Content-Type": "image/png"}
+
+
+def test_parse_meta_service():
+    parsed = parser._parse((200, json.dumps({"Content-Type": "application/pdf"})), service="meta")
+    assert parsed == {"metadata": {"Content-Type": "application/pdf"}, "content": None, "status": 200}
+
+
+def test_parse_text_service_returns_body_verbatim():
+    parsed = parser._parse((200, "plain text"), service="text")
+    assert parsed == {"metadata": None, "content": "plain text", "status": 200}
+
+
+def test_parse_empty_body():
+    assert parser._parse((204, "")) == {"metadata": None, "content": None, "status": 204}
+    assert parser._parse(None) == {"metadata": None, "content": None}
+
+
+def test_parse_does_not_decode_error_bodies():
+    parsed = parser._parse((503, json.dumps({"status": "TIMEOUT", "message": "worker timed out"})))
+    assert parsed == {"metadata": None, "content": None, "status": 503}
+    parsed = parser._parse((422, "Unprocessable Entity"), service="meta")
+    assert parsed == {"metadata": None, "content": None, "status": 422}
+
+
+# --- integration, against the live server -----------------------------------------
+
+
 def test_remote_pdf(remote_fixture_base_url):
     """parse remote PDF"""
     assert parser.from_file(f"{remote_fixture_base_url}/remote.pdf")