Starting this week blog posts are archived in the Rogue Scholar science blog archive using a dedicated invenio module instead of an external service. This simplifies the maintenance of the service and is critical for the long-term future of the archived content.

Rogue Scholar started out as a bespoke service written in Javascript in early 2023. In September 2024 Rogue Scholar started the migration to the InvenioRDM repository platform. This migration is now complete with the release of the invenio-feeds module that provides functionality to automatically parse blog feeds and archive the metadata and content in a digital repository. This extends functionality already available in InvenioRDM, or added in previous work – DOI registration with Crossref, full-text search, and automated subject classification with the OpenAlex vocabulary falls in the later category.

Until the release of invenio-feeds, automatic parsing of newly published or updated science blog posts was done by a dedicated Javascript and later Python service hosted at api.rogue-scholar.org. This service is no longer needed and will be retired on November 1st, reducing the complexity and cost of Rogue Scholar infrastructure. One example is the use of GitHub Actions to trigger the parsing of all blog feeds every 10 min which is now handled by InvenioRDM background workers.

The switch from dedicated external service to Python module has been fairly smooth so far, but I absolutely expect issues to come up in the coming weeks and months.

The migration to invenio module not only simplifies Rogue Scholar infrastructure, but comes with new functionality: archiving of blog posts as PDF files in the PDF/A format as InvenioRDM file attachments.

One fundamental archiving principle is LOCKSS (Lots of Copies Keep Stuff Safe), and these PDF files nicely complement archiving activities by the blog itself, archiving of metadata and full-text in Rogue Scholar, and archiving of participating science blogs in the Internet Archive Archive-It service.

These PDF files are generated the moment content is added or updated to Rogue Scholar, and it will take a few months to address issues with the PDF files (e.g. pagination, image sizing), and archiving the more than 50K science blog posts in Rogue Scholar.

The PDF files use the same layout as the PDF files generated by Rogue Scholar since January 2024, but are generated with every submission to Rogue Scholar instead of dynamically on demand (using the commonmeta-py library). The PDF files are again generated with the WeasyPrint and pikepdf Python libraries, but no longer use Pandoc and the markdown format as an intermediary step. Other formats (markdown, ePub or JATS XML) are no longer supported, and this simplification makes the PDF generation workflow simpler and faster – allowing me to generate more than 3000 PDF attachments this week.

I have a painful relationship with PDF and scholarly publishing, and have long thought the proper archiving format for Rogue Scholar. WARC is a widely used archiving format developed and used by the Internet Archive – supported in InvenioRDM since the recent v14 release. Markdown and ePub are open formats closer to the technologies used by blogging platforms. PDF is fine for reading and archiving, but very painful to work with to extract content out again. The decision for PDF/A (and not WARC or ePub) was mainly for two reasons:

  • a widely supported official archiving format (e.g. digital repositories),
  • widely used by scholars and supported by the tools and workflow they use.

The PDF/A that Rogue Scholar generates has two important features that overcome critical PDF shortcomings:

  • rich metadata in XMP format, including DOI, authors, title, abstract and license
  • full-text content in HTML format as attachment that can easily be accessed, using the PDF/A-3a standard.

Over the coming months I will not only work on any issues that come up with the new invenio-feeds module and PDF generation, but reach out to participating science blogs about archiving the PDF/A files in associated repositories, starting with the Invenio blog and Zenodo.

Please reach out via Slack, email, Mastodon, or Bluesky if you have any questions or comments.

Rogue Scholar is a scholarly infrastructure that is free for all authors and readers. You can support Rogue Scholar with a one-time or recurring donation or by becoming a sponsor.

References

  1. Fenner, M. (2022, December 12). Building an archive for scholarly blog posts. Front Matter. https://doi.org/10.53731/br9f5xa-a556w2t
  2. Fenner, M. (2024, September 2). Rogue Scholar migrates to InvenioRDM. Front Matter. https://doi.org/10.53731/sdazp-kzn55
  3. Fenner, M. (2023, October 30). Starting November, all Rogue Scholar blog posts will be archived by the Internet Archive. Front Matter. https://doi.org/10.53731/hhtx0-wb293
  4. Fenner, M. (2024, January 8). Every Rogue Scholar blog post now available in Markdown, ePub, and PDF formats. Front Matter. https://doi.org/10.53731/1dfxr-hs665
  5. Fenner, M. (2010, October 6). Beyond the PDF – it is time for a workshop. Front Matter. https://doi.org/10.53731/r294649-6f79289-8cw7z