OpenAPC: Technical Background

Christoph Broschinski, Bielefeld University Library, <broschinski@uni-bielefeld.de>

This presentation is also available at http://www.ub.uni-bielefeld.de/~cbroschinski

Timeline

OpenAPC: Started in 2014 as a private endeavour by our colleague Najko Jahn (Project Manager at Bielefeld University Library)

First Participants: German University Libraries receiving funding for paying APCs from the Open Access Publishing Programme set up by the DFG (German Research Foundation)

Good opportunity to start an open data project on APCs:

Annual reports required from funded libraries -> Compiled APC data already existing
Third-party funding -> Less reservations on sharing financial data

2015: Application for DFG funding together with 2 partnering initiatives:

ESAC at the Max Planck Digital Library (MPDL), Munich (workflows and management connected to OA publishing)
OA Analytics at the Institute for Interdisciplinary Studies of Science (I²SoS), Bielefeld University (OA bibliometrics)

Funding granted (for 3 years, starting in October 2015), partners teamed up under new label INTACT

Data Storage and Schema

The Open APC data is stored and maintained in a core data file on GitHub ("data/apc_de.csv")
Core data file format is described by the Open APC Schema definition.

column	description	source	input_required
institution	Top-level organisation which covered the fee, e.g. Bielefeld University	none	mandatory
period	Year of APC payment (YYYY)	none	mandatory
euro	The amount that was paid in EURO. Includes VAT and additional fees	none	mandatory
doi	Digital Object Identifier	none	mandatory
is_hybrid	Has the article been published in a toll access journal?	none	mandatory
publisher	Name of publication house that has charged the fee	CrossRef	optional
journal_full_title	Full name of periodical that contains the article	CrossRef	optional
issn	International Standard Serial Number. If more than one are available, collapse with `;`	CrossRef	optional
issn_print	International Standard Serial Number - print version	CrossRef	no
issn_electronic	International Standard Serial Number - electronic version	CrossRef	no
license_ref	License under which the research paper has been published	CrossRef	no
indexed_in_crossref	checks if the contribution is registered with the DOI agency CrossRef	CrossRef	no
pmid	id for metadata records indexed in Europe Pubmed Central (Europe PMC)	Europe PMC	no
pmcid	id for articles available in Europe PubMed Central full text collection	Europe PMC	no
ut	Web of Science unique item id	Web of Science	no
url	URL to article if no DOI is available	none	optional
doaj	Is the journal indexed in the Directory of Open Access Journals (DOAJ) ?	DOAJ	no

Institutions are required to follow the schema when contributing data. However:

Only the 5 mandatory fields are always required
The 4 optional fields are required if (and only if) the article in question does not have a DOI (rarely the case in 2016!)
Every other field can (and should) be omitted, because these values are determined automatically by our metadata enrichment script
Consequence: Some submitted data tables do consist of only 5 columns!

Open APC schema "philosophy": Keep the set of mandatory fields as small as possible and infer as much data as possible automatically from external services!

Every additional mandatory field puts more of a burden on contributing institutions (Increased workload for them)
Every additional mandatory field introduces another source of mistakes (Fields missing, fields badly formatted, checking back required, increased workload for us and them.)
Every additional mandatory field introduces another source of data inconsistency (Spelling of journal titles, publisher names)

Data Submission Workflow

1) Institutions may submit their data in one of two ways:

Via E-Mail (Attached CSV file or Excel/LibreOffice Spreadsheet)
Via GitHub (By putting a CSV file into their institutional directory and making a pull request)

2) Submitted files will usually require editing in Excel for minor corrections (Institution abbrevations, currency format, column header names)

3) Files are (re)exported to CSV format

4) Files are processed by our automatic metadata enrichment script

Python script, located at python/apc_csv_processing.py
Input: CSV file with at least the 5 mandatory OpenAPC Columns
Output: Fully enriched 17-column CSV file (metadata from crossref, pubmed and DOAJ), conforming to the Open APC schema
Script can handle many inconsistencies on its own (wrong column order and names, file encodings...) and can otherwise be fine-tunded with command line parameters

5) Contents of script output file can be directly copied into Open APC core data file!

Report Generation

A general overview over the OpenAPC data is given on the GitHub front page (README.md) with tables and plots
Problem: OpenAPC core data file changes quite regularly - how to keep all statistics up-to-date without too much effort?
Solution: Automatic Generation of README.md from an .Rmd ("R Markdown") template
Very easy handling: Whenever the core file changes, call R -e "knitr::knit('README.Rmd')" from the Open APC main directory.

The same principle is used to generate the blog posts on our github.io-Blog

Automated integrity testing

Letting a script generate the contents of the Open APC core data file will ensure its syntactic correctness. But what about semantic correctness?
For example:

Duplicates (Same DOI occurs more than once)
Name inconsistencies (Same ISSN, but different journal/publisher name)
Invalid column values (Example: is_hybrid should either be TRUE oder FALSE, nothing else)
Minimal data requirements not met (Example: No DOI given and not all 4 alternative columns filled out)

Solution: Python-based testing script which will check the core data for these problems.

Script is integrated into Travis, a free Continuous integration (CI) Service. Whenever a push or pull-requested is made to the Open APC repository, Travis automatically invokes the testing script and checks the core data file for semantic problems.

Advanced Visualisation and Analysis

Problem 1: The Open APC core data file is difficult to analyse without tool support.

Solution: An OLAP (Online Analytical Processing) server was set up to allow more easy querying of OpenAPC data.

Example: http://olap.intact-project.org/cube/openapc/aggregate?drilldown=publisher&cut=period:2014 (Meaning: "How much money did each individual publisher receive in 2014?")

Problem 2: The Open APC data is hard to explore / there are only static visualisations

Solution: The OLAP server is used as a backend for a treemap visualisation project

So I heard you want to start your own Open APC project...

Recommendations:

0. Have an initial group of institutions join and acquire some testing data to start with (Done)

1. Simply copy (most of) the structure and files of our GitHub repository

No need to reinvent the wheel
The more compatible our projects stay, the easier it will be to share data and support each other later

2. Define a data schema

Same recommendation: Use ours with only minimal adaptions to swedish conditions (currency?)

3. Familiarise yourselves with the processing script, enrich some data and build an initial core data file

Requires a python installation

4. Adapt the README.Rmd template and use it to generate a custom REAMDE.md file from your data.

Requires an R installation with some additional packages (knitr etc.)

Questions?

Otherwise: Thanks for your attention...

...and let's get to work!