Docs/diataxis restructure

.gitignore

@@ -66,6 +66,9 @@ instance/
 documentation/_*/
 documentation/source/
 
+# MkDocs documentation
+site/
+
 # PyBuilder
 target/
 

documentation/explanation/facades.md

@@ -0,0 +1,14 @@
+# Default Configuration for Facades
+
+The easiest way to get started is to use the facades and configuration defaults. The
+facades set up both logging and HDX configuration.
+
+The default configuration loads an internal HDX configuration located within the
+library, and assumes that there is an API key file called **.hdxkey** in the current
+user's home directory **\~** and a YAML project configuration located relative to your
+working directory at **config/project_configuration.yaml** which you must create. The
+project configuration is used for any configuration specific to your project.
+
+The default logging configuration reads a configuration file internal to the library
+that sets up an coloured console handler outputting at INFO level and a file handler
+writing to errors.log at ERROR level.

documentation/explanation/history.md

@@ -0,0 +1,18 @@
+# Brief History
+
+The first iteration of a scraper for ACLED was written without the HDX Python library
+and it became clear looking at this and previous work by others that there are operations
+that are frequently required and which add unnecessary complexity to the task of coding
+against HDX. Simplifying the interface to HDX drove the development of the Python
+library and the second iteration of the scraper was built using it. ACLED went from
+producing files to creating an API, so a third iteration was developed.
+
+With the interface using HDX terminology and mapping directly on to datasets, resources
+and showcases, the ACLED scraper was faster to develop and was much easier to understand
+for someone inexperienced in how it works and what it is doing. The extensive logging
+and transparent communication of errors is invaluable and enables action to be taken to
+resolve issues as quickly as possible. Static metadata can be held in human readable
+files so if it needs to be modified, it is straightforward.
+
+The HDX Python library has expanded over time as needs have arisen and is used for a
+range of tasks involving interaction with the HDX platform.

documentation/explanation/information.md

@@ -0,0 +1,11 @@
+# Information
+
+For more about the purpose and design philosophy, please visit
+[HDX Python Library](https://humanitarian.atlassian.net/wiki/display/HDX/HDX+Python+Library).
+
+This library is part of the [Humanitarian Data Exchange](https://data.humdata.org/)
+(HDX) project. If you have humanitarian related data, please upload your datasets to
+HDX.
+
+The code for the library is [here](https://github.com/OCHA-DAP/hdx-python-api).
+The library has detailed API documentation which can be found in the menu at the top.

documentation/how-to/activity-streams.md

@@ -0,0 +1,33 @@
+# Use Activity Streams
+
+The **Activity** class wraps CKAN activities, which record who changed what and when.
+Activities are read only. You can get the activity stream of a dataset, organization,
+location or user:
+
+    activities = dataset.get_activities(**kwargs)
+    activities = organization.get_activities(**kwargs)
+    activities = location.get_activities(**kwargs)
+    activities = user.get_activities(**kwargs)
+
+These return a list of **Activity** objects, each with fields like `timestamp`,
+`user_id`, `object_id` and `activity_type` (eg. "changed package"). One page of results
+is returned. Its size is set by `limit` (default 31, HDX maximum 100) and where it starts
+by `offset`. Passing `all_pages=True` gets every page. Other arguments (`**kwargs`) like
+`after`, `before` and `activity_types` depend on the object type and are detailed in the
+API documentation. Getting a user's activity stream requires authorisation to view the
+user's profile.
+
+You can get the activity stream of all recently added or changed datasets:
+
+    activities = Activity.get_recently_changed_packages(limit=10)
+
+With `all_pages=True`, this stream covers the whole history of HDX so it is capped by
+`max_activities` (default 1000). Pass `max_activities=None` to remove the cap.
+
+You can read a single activity, get its data (including a snapshot of the object at the
+time of the activity) or get a diff against the object's previous version. HDX requires
+permission to edit the object the activity relates to for these:
+
+    activity = Activity.read_from_hdx("ACTIVITY_ID")
+    data = activity.get_data()
+    diff = activity.get_diff(diff_type="unified")  # or "context" or "html"

documentation/how-to/api-key.md

@@ -0,0 +1,24 @@
+# Obtain your API Key
+
+If you just want to read data from HDX, then an API key is not necessary and you can
+ignore the 6 steps below. However, if you want to write data to HDX, then you need to
+register on the website to obtain an API key. You can supply this key as an argument or
+create an API key file. If you create an API key file, by default this is assumed to be
+called **.hdxkey** and is located in the current user's home directory **\~**. Assuming
+you are using a desktop browser, the API key is obtained by:
+
+1. Browse to the [HDX website](https://data.humdata.org/)
+2. Left click on LOG IN in the top right of the web page if not logged in and log in
+3. Left click on your username in the top right of the web page and select PROFILE from
+the drop down menu
+4. Scroll down to the bottom of the profile page
+5. Copy the API key which will be of the form: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
+6. You can either:
+
+    a.  Pass this key as a parameter or within a dictionary
+
+    b.  Create a JSON or YAML file. The default path is
+            **.hdx_configuration.yaml** in the current user's home
+            directory. Then put in the YAML file:
+
+            hdx_key: "HDX API KEY"

documentation/how-to/customisation.md

@@ -0,0 +1,60 @@
+# Customise the Configuration
+
+It is necessary to pass configuration parameters in the facade call eg.
+
+    facade(main, user_agent=USER_AGENT, hdx_site = HDX_SITE_TO_USE, hdx_read_only = ONLY_READ_NOT_WRITE, hdx_key_file = LOCATION_OF_HDX_KEY_FILE, hdx_config_yaml=PATH_TO_HDX_YAML_CONFIGURATION, project_config_dict = {"MY_PARAMETER", "MY_VALUE"})
+
+If you do not use the facade, you can use the **create** method of the **Configuration** class directly, passing in
+appropriate keyword arguments ie.
+
+    from hdx.api.configuration import Configuration
+    ...
+    Configuration.create([configuration], [user_agent], [user_agent_config_yaml], [remoteckan], KEYWORD ARGUMENTS)
+
+You must supply a user agent using one of the following approaches:
+
+1. Populate parameter **user_agent** (which should be the name of your
+organisation and project)
+2. Supply **user_agent_config_yaml** which should point to a YAML file which
+contains a parameter **user_agent**
+3. Supply **user_agent_config_yaml** which should point to a YAML file and populate
+**user_agent_lookup** which is a key to look up in the YAML file which should
+be of form:
+
+        myproject:
+            user_agent: test
+        myproject2:
+            user_agent: test2
+
+4. Include **user_agent** in one of the configuration dictionaries or files outlined in
+the table below eg.
+    **hdx_config_json** or **project_config_dict**.
+
+**KEYWORD ARGUMENTS** can be:
+
+|Choose|Argument|Type|Value|Default|
+|---|---|---|---|---|
+| |hdx_site|Optional\[str\]|HDX site to use eg. prod, feature|test|
+| |hdx_read_only|bool|Read only or read/write access to HDX|False|
+| |hdx_key|Optional\[str\]|HDX key (not needed for read only)||
+|Above or one of:|hdx_config_dict|dict|Dictionary with hdx_site, hdx_read_only, hdx_key||
+|or|hdx_config_json|str|Path to JSON configuration with values as above||
+|or|hdx_config_yaml|str|Path to YAML configuration with values as above||
+|Zero or one of:|project_config_dict|dict|Project specific configuration dictionary||
+|or|project_config_json|str|Path to JSON Project||
+
+To access the configuration, you use the **read** method of the **Configuration** class as follows:
+
+    Configuration.read()
+
+For more advanced users, there are methods to allow you to pass in your own
+configuration object, remote CKAN object and list of valid locations. See the API
+documentation for more information.
+
+This global configuration is used by default by the library but can be replaced by
+Configuration instances passed to the constructors of HDX objects like Dataset eg.
+
+    configuration = Configuration(KEYWORD ARGUMENTS)
+    configuration.setup_remoteckan(REMOTE CKAN OBJECT)
+    configuration.setup_validlocations(LIST OF VALID LOCATIONS)
+    dataset = Dataset(configuration=configuration)

documentation/how-to/docker.md

@@ -0,0 +1,6 @@
+# Build a Docker image
+
+The library is also available set up and ready to go in a Docker image:
+
+    docker pull public.ecr.aws/unocha/hdx-scraper-baseimage:stable
+    docker run -i -t public.ecr.aws/unocha/hdx-scraper-baseimage:stable python3

documentation/how-to/facades.md

@@ -0,0 +1,42 @@
+# Use Facades
+
+The simple facade makes it easier to get up and running:
+
+    from hdx.facades.simple import facade
+
+    def main():
+        ***YOUR CODE HERE***
+
+    if __name__ == "__main__":
+        facade(main, CONFIGURATION_KWARGS)
+
+The keyword arguments facade is similar but passes through keyword arguments:
+
+    from hdx.facades.keyword_arguments import facade
+
+    def main(kwparam1, kwparam2, ...,**ignore):
+        ***YOUR CODE HERE***
+
+    if __name__ == "__main__":
+        facade(main, CONFIGURATION_AND_OTHER_KWARGS)
+
+The infer arguments facade infers the possible command line from the type hints and
+docstring describing the parameters of the function you give (such as main). It is
+possible to pass additional parameters to the facade which will be overridden by any
+parameters provided on the command line with the same name.
+
+    from hdx.facades.infer_arguments import facade
+
+    def main(kwparam1: bool, kwparam2: str):
+        """Generate dataset and create it in HDX
+        Args:
+            kwparam1 (bool): Help text for this command line argument
+            kwparam2 (str): Help text for this command line argument
+
+        Returns:
+            None
+        """
+        ***YOUR CODE HERE***
+
+    if __name__ == "__main__":
+        facade(main, kwparam3="lala")

documentation/how-to/framework.md

@@ -0,0 +1,57 @@
+# Build a project Framework
+
+Once you understand how to create a dataset on HDX, it is important to think
+about a good structure. Below is framework for starting a project to interact
+with HDX that should work well.
+
+First, pip install the library or alternatively add it to a requirements.txt
+file if you are comfortable with doing so as described in [Install the Library](install.md).
+
+Next create a file called **run.py** and copy into it the code below.
+
+    #!/usr/bin/python
+    # -*- coding: utf-8 -*-
+    """
+    Calls a function that generates a dataset and creates it in HDX.
+
+    """
+    import logging
+    from hdx.facades.simple import facade
+    from .my_code import generate_dataset
+
+    logger = logging.getLogger(__name__)
+
+
+    def main():
+        """Generate dataset and create it in HDX"""
+
+        dataset = generate_dataset()
+        dataset.create_in_hdx()
+
+    if __name__ == "__main__":
+        facade(main, hdx_site="test")
+
+The above file will create in HDX a dataset generated by a function called
+**generate_dataset** that can be found in the file **my_code.py** which we will now
+write.
+
+Create a file **my_code.py** and copy into it the code below:
+
+    #!/usr/bin/python
+    # -*- coding: utf-8 -*-
+    """
+    Generate a dataset
+
+    """
+    import logging
+    from hdx.data.dataset import Dataset
+
+    logger = logging.getLogger(__name__)
+
+
+    def generate_dataset():
+        """Create a dataset
+        """
+        logger.debug("Generating dataset!")
+
+You can then fill out the function **generate_dataset** as required.

documentation/how-to/idmc-example.md

@@ -0,0 +1,14 @@
+# IDMC Example
+
+A complete example can be found here: <https://github.com/OCHA-DAP/hdx-scraper-idmc>
+
+The IDMC scraper creates a dataset per country in HDX, populating all the
+required metadata. It then creates resources with files held on the HDX
+filestore.
+
+In particular, take a look at the files **run.py**, **idmc.py** and the
+**config** folder. Do not run it unchanged as it may overwrite the existing
+datasets in the IDMC organisation (although it will most probably fail as you
+will not have permissions to modify anything in that organisation). You can use
+it as a basis for your code renaming and modifying **idmc.py** as needed and
+updating metadata in **config/hdx_dataset_static.yaml** appropriately.

documentation/how-to/install.md

@@ -0,0 +1,18 @@
+# Install the Library
+
+To include the HDX Python library in your project, you must **pip install** or add to
+your **requirements.txt** file the following line:
+
+    hdx-python-api==VERSION
+
+Replace **VERSION** with the latest tag available from
+<https://github.com/OCHA-DAP/hdx-python-api/tags>.
+
+If you get dependency errors, it is probably the dependencies of the cryptography
+package that are missing eg. for Ubuntu: python-dev, libffi-dev and libssl-dev. See
+[cryptography dependencies](https://cryptography.io/en/latest/installation/#building-cryptography-on-linux).
+
+If you get import or other errors, then please either recreate your virtualenv if you
+are using one or uninstall hdx-python-api, hdx-python-country and hdx-python-utilities
+using **pip uninstall**, then install hdx-python-api (which will pull in the other
+dependencies).

documentation/how-to/locations.md

@@ -0,0 +1,52 @@
+# Manage Locations
+
+The **Location** class wraps HDX locations (CKAN groups eg. countries, "world"). Reading
+locations does not require any special permissions, so these are the operations you will
+typically use. Creating, updating and deleting locations is restricted to HDX sysadmins
+and works the same way as for other HDX objects (see
+[Use HDX Objects](objects.md)).
+
+You can read a single location:
+
+    location = Location.read_from_hdx("LOCATION_ID_OR_NAME")
+
+You can get all location names in HDX:
+
+    location_names = Location.get_all_location_names(**kwargs)
+
+Passing `all_fields=True` (plus optionally `include_extras=True` etc.) returns the full
+location dictionaries instead of just names. Various additional arguments (`**kwargs`)
+can be supplied and are detailed in the API documentation.
+
+You can fetch the countries that are currently active on HDX's Data Grid (a live lookup,
+since Data Grid membership changes over time, rather than a fixed list):
+
+    country_codes = Location.get_data_grid_countries()
+
+This returns a sorted list of 3-letter location codes.
+
+HDX sysadmins can get the Data Grid of a country and clear HDX's cached copy of
+it:
+
+    data_grid = Location.get_data_grid("AFG")
+    Location.invalidate_data_grid("AFG")
+
+**get_data_grid** returns a dictionary with keys including `title`, `date`,
+`stats` and `categories` (or None if not found). Each category has
+`subcategories`, each with a list of `datasets`. Complementary datasets have
+`is_complementary` set to True.
+
+You can get the datasets belonging to a location as follows:
+
+    datasets = location.get_datasets(**kwargs)
+
+Various additional arguments (`**kwargs`) can be supplied. These are detailed in the API
+documentation.
+
+You can autocomplete a location name:
+
+    matches = Location.autocomplete("NAME")
+
+For simple name/code lookups against a cached list of valid locations (eg. converting
+between a country name and its HDX code) rather than the full **Location** object, see
+the **Locations** helper class instead.