Posts on Joe Carrano

Transfer Metrics Needed Tracking (TMNT)

Wed, 27 Sep 2023 08:42:40 -0400

This (fiscal) year, I’m taking some next steps using the data produced by my digital media survey at MIT. Now that we have a record of what media exist in the collections to the best of our knowledge, we can begin transferring the data off of them at a larger scale and into preservation and access storage. But wait… could I use this work as a representative sample to better plan for the future? Enter the Transfer Metrics Needed Tracking (TMNT) project.

Props to my colleague Chris Tanguay for making me aware of the TMNT logo generator website

What is it?

This is a project to record some basic numbers about the time and success rate of data transfer from digital media over the next year. This is similar to what other institutions have done, such as the British Library’s Flashback project.

For this project I’ve imported the media survey data into Airtable and linked it up with another base where we’ll use an interface to record the following info:

Setup time
Transfer time and attempts
Transfer method
Overall success/failure
Extraction time and attempts (if not performed in the transfer process)
Packaging time and attempts (if not performed in the transfer process)
Data size
Computer used

The information we gather will be enhanced in the final data set by contextual information we already know from the survey: format types and media manufacturer information. The details around actually processing the files will be recorded with the rest of our processing metrics.

Why are you doing this??

Really this is for planning purposes. We need to know about how long these projects might take in order to allocate enough time and person power to get the work done while balancing many other priorities. The project isn’t a way to establish benchmarks that need to be met in some sort of Taylorist archival transfer process nightmare scenario.

With a decent amount of a digital media backlog in existing collections, there will be many projects to transfer the data to a more stable medium over the coming years. At the same time, we’ll have new collections coming in. With a better sense of how much work we have in store, we can also make more informed decisions on whether we have the capacity to accept collections with substantial amounts of digital media or respectfully decline.

My hope is that this information will be useful to others as well, so I’ll to share my results out at the end and possibly along the way. Maybe somewhere else will do a similar project and we can compare numbers? In any event, grab a slice of pizza and check back for updates.

Digital media survey results

Wed, 03 May 2023 10:11:38 -0500

Over the last year, I’ve worked on a digital media survey of our archival and distinctive collections at MIT Libraries. I mentioned in that post linked above I would update this blog along the way but that didn’t really happen, sorry friends!

Media circus

Things went pretty well and a bit faster than I expected. It took me just over a year with some fits and starts to finish the whole survey process. The first stage was our archival and manuscript collections, of which, I was able to review over 300 boxes and a few dozen rare books and publications in 8 months.

Along the way, I transferred data from media we could in handle in house within collections that had a total of 2 or fewer pieces of media since it was convenient while the boxes were onsite. The small number ensured that we could get them done relatively easily and have a quick win of crossing a collection off the list with media needing to transfer data from.

After surveying the first stage of surveying, I started on the academic theses, which are also under our purview. The work to assemble the list of media from published material was started earlier when needing to find this information for publications and rare books in the main collections mentioned above. As these material have their metadata stored as MARC records in Alma, I went about putting together a similar list of material as with the archival records. Within Alma, I ran a number of “indication rules” looking in the 300 field, subfields a and e for the terms: dis*, flopp*, zip,computer, cartridge, CD*, DVD*, USB, and drive. Additionally, local cataloging practice for a time was to use a call number prefix for media, often when those items were included with another print title with the same call number. I was able to assemble a list of these prefixes like CDROM, DSKETTE, etc. and search for them in Alma as well, in case this surfaced anything missed in the 300 field search. I combined the results into a spreadsheet, deduped them, and then got to work surveying and storing results in our media log tool.

Work on the theses took about another 4 months as getting them from offsite storage was just as time consuming as when doing so with the main part of the survey. With some of the these stored in boxes as a group, I often had to request many boxes just to get through a year with a few theses that contained external media. Ultimately, I was able to finish up everything by the end of February of this year.

Results

Here are the results from our the total survey (including theses). I’ve simplified CDs and DVDs to be a summary of all of their subtypes.

Format	Count
Floppy disk: 3.5 inch	909
CD	846
Floppy disk: 5.25 inch	394
DVD	339
Data tape	278
Data cartridge	238
Zip disk	126
Floppy disk: 8 inch	49
Jaz disk	7
Hard drive	4
Computer	4
Flash memory: USB	3

Overall, I found just slightly under 3000 pieces of digital media in our distinctive collections and another 250ish pieces of media included with theses. It’s thankfully much less than you might expect for a technology-focused place like MIT. The majority of media found were floppy and optical formats that we have established workflows and equipment to handle. Whether the content is readable or not is another story but we can make an attempt.

While we can handle most of the found formats, the survey revealed a sizable number of 8-inch floppy disks, data cartridges, and mainframe data tapes, that we don’t have an in-house solution for. Those results are concerning but this is exactly what we wanted to surface for planning purposes.

Lessons Learned

Things didn’t always go smoothly, this survey ended up being an exercise in metadata remediation. We often encountered errors in box numbers, missing items, total lack of description, and completely misidentified things.

One particular frustration with the archival and manuscript collections was that 180 entries out of 640 turned out to be false positive matches for digital media. About half of those were from using the phrase “backup” or “back-up”, which was a false match 92 percent of the time. Often it really just meant paper backups. It turns out that despite seeing the words “backup” written on labels of a lot of the media I found, it was almost never entered into a finding aid in those cases.

Future steps

This data has already been a great starting point for better understanding how much work is ahead in analyzing and transferring digital media and how we can plan to go about it. We’ve started dissecting the specifics of the data to prioritize some of it, such as institutional records we have a clear mandate to preserve and things from women and BIPOC related collections. In the next year, we’ll begin more work on getting though the media as well as establishing metrics on how long on average it takes to process different media types. This will help give us a better idea how long we will need to commit to some of the transfer projects.

Distressed Links

Wed, 13 Jul 2022 08:50:02 -0400

At work I’ve been crawling some form of the MIT COVID-19 response webpage daily since early March 2020. In that time there have been a couple of domain name switches but it’s been pretty boring considering the scope and tragedy of the ongoing pandemic.

In June, I was reviewing some of our crawl reports for this seed and I noticed an unusual url in the host report: archive.org, i.e. the big brother of Archive-It, THE (with no respect to Ohio State) Internet Archive. Obviously, this shouldn’t happen unless the site was actually linking out to IA.

After some investigation, I found that there were a few links to IA! It’s web archives all the way down. The strange thing was that the links were on the MIT Travel policy page, pointing to information from the Department of State and the CDC (under “Additional Resources”) that is still actively maintained and people would need the most up-to-date information about. Instead, they were getting pandemic related travel info from late March!

Checking out the Archive-it Wayback captures for the MIT travel page, I saw that this change happened around May 11. You can see the proper links on May 9 but then those links are removed on May 11 and then added back in later in the day, but replaced with IA Wayback links. Weird! For their part, I was able to find the maintainers of the website and they fixed the links pretty quickly.

They didn’t tell me how this might of occurred but we can speculate a couple scenarios:

Somebody accidently deleted those links when updating the website and had to go into IA out of ease or necessity to find what was there before. They then did a simple copy+paste instead of editing out the Wayback url prefix. In any event, maintain backups!
Or some link checking plug-in went awry and inserted a Wayback link in place of the original links, due to an error saying the page was no longer available (temporary 404, etc.).

Either way, the result was that the content was made to look outdated or old, kind of like how you can distress a piece of furniture to make it look vintage. I’m not about to yuck anyone’s yum but neither of those things are for me. While there was no ill intent here, you can see how easy it is to spread misinformation this way, especially if people aren’t looking closely. The pandemic isn’t over and people need up-to-date information to inform their decisions.

Link rot is a real problem and web archives have a place to play in mitigating that in information resources. However, current information shouldn’t be replaced by a snapshot of something in time. If we’re going to insert archived links into web pages either manually or via automation, the solutions to display them are already out there such as robust links, where you can display both a current link and a web archived version very easily. You’ve then got the up-to-date info and a web archived link just in case. Make your links robust, not distressed!

Learning to walk again: starting a digital media survey

Tue, 25 Jan 2022 20:45:52 -0500

In Ricky Erway’s foundational instructional guide, “You’ve Got to Walk Before You Can Run: First Steps for Managing Born-Digital Content Received on Physical Media”, we learn that “walk” in this context means to know about the physical media that you have in your collection, the where, the how much, and the what type. This is also related to the archival principles of physical and intellectual control of collections. This report inspired many archival institutions to begin surveying their collections for born-digital media as a basic starting point to approaching born-digital processing. The trend included multiple rounds of SAA’s “Jump-in” initiative which encouraged repositories to do surveys and share their results.

Where I work, the MIT Libraries, for various reasons, did not do a survey or partake in the Jump-in initiatives. There were efforts to survey some collections but there wasn’t anything comprehensive. At the same time, there was some transfer of material off of digital media found in collections. So we weren’t walking or jumping but we would do random sprints and then collapse into a crawl, gasping for breath while working on other projects that came along.

In recent years, I’ve worked to formalize our procedures for transferring content off of born-digital physical media and I feel like we’re in a good place to do that work for common media types. In 2019, I began the start of work for a media survey but was delayed due to the pandemic (surprise!). When I regained the ability to access the collections again in fall 2021, I began the survey work in earnest.

Goals

In starting this work I have a few goals for what we hope to get out of it. As you might expect, I want to figure out what media we have and where it is! Beyond this simple knowledge, doing this work will give us a more holistic understanding of the media in our collection that can inform our future processing efforts, appraisal, staffing, and what equipment we may need. While all of this media is at some level of risk remaining stored in these formats, knowing the collections that contain the media can help us take an EDISJ approach in prioritizing the transfer of content by and about white women and people of color of all genders.

Looking for leads

We started the survey process by developing a list of terms related to digital media, which we could then search for in our ArchivesSpace repository and create a report from the API with: Floppy, Floppies, Disc, Discs, Disk, Disks, Diskette, Diskettes, CD, CDs, CD-rom, CD-roms, DVD, DVDs, USB, Hard drive, Flash drive, Digital, Zip, Data tape, Data tapes, Thumb drive, Backup, Backups, Cartridge, Cartridges.

Using this list we ended up with a pretty long CSV of possible elements within finding aids and accessions to investigate. To make this list a little more manageable, I separated out the terms that were the result of just searching “digital” to possibly work on at a later stage of the process, as these would most like contain the most false positives.

I started going through the remaining list for any false positives or duplicate entries. We ended up with 545 entries, which isn’t that solid of a number as in many cases I can’t tell whether description is in aggregate or an item level. I used formulas to total up the frequency per search term and the number of results per archival collection which I can use as a starting point to figure out which collections to call first. I then took the paired down list and bulk imported them into an Aeon “activity” for eventually requesting their boxes from storage.

How to record survey results

In planning for this survey we needed a method to record our results once the boxes are called from storage and we get that sweet plastic, magnetic, and optical substrate into our hands. That’s where the media log will come in!

Archivists at other institutions developed multiple media logs (NYU and Rockefeller Archive Center, for instance) for similar situations as us, recording the presence of digital media in collections, it’s type, and whether the content has been migrated to a more accessible medium. These other instances caught the eye of staff members here years ago and we had a version that was spun up for us (but rewritten in Java, not sure why…). The app sat largely unused since that time, when the pandemic started it was fully taken offline due to a need of a infrastructure migration and the realization that it wouldn’t be used for some time.

As part of the survey prep we decided to get the app running again because it seemed like it would fit our needs. This took a little while with some support from staff in digital library services but we’re back up and running. We made some minor additions for things that might be useful like label info and categories of media that weren’t present but the app largely remains similar.

Next steps

With all this geared up, we were hoping to get started with actually doing the survey at the start of this new year but with the pandemic raging I’ve been fully working from home this past month. Delayed again! But what can you do? Before going remote again, I already had done some calculations based on the number of boxes I’m allowed to call to campus (we use offsite storage), the number of days I work on campus during a week, and the time to do the survey against the time remaining and it would be impossible to finish by the end of the fiscal year (when I wanted to complete the survey).

So even with the month delay, things were already a bit behind. I’m ready to push ahead with the actual survey regardless of the time it takes and hope to share some more updates along the way!

A short look @ Kids dot US, a failed public/private web experiment [for kids]

Tue, 24 Aug 2021 20:26:29 -0400

A little while ago I went down a Wikipedia rabbit hole and ended up at the the Wikipedia article for the .us domain. I was particularly intrigued by the section for Kids.us. Have you heard of it? I know I had no idea what it was.

As the Wikipedia article states, it was a subdomain of the .us domain made specifically for kids and created by U.S. law! Through an act of Congress in 2002, lawmakers created the subdomain to be used by content providers to create kid-centric websites. It never really took off and by the end there were only 6 active websites left on the subdomain: Nick, NickJr, PBSkids, Smithsonian, and two identical sites selling trampolines.¹ ²

This peaked my interest. What were these websites like? Was there any more information out there? Why did the U.S. government ever think this was a good idea?

Birthing a subdomain

In the early 2000s, just as today, politicians were concerned about kids on the internet. Especially how sites were using common word web domains to host pornography, which many people, including children, would stumble upon accidentally. A bipartisan group of lawmakers felt the solution would be to create a domain where there would only be appropriate content for children, which many of them described akin to the “children’s section of a library” but of course probably never spoke to a children’s librarian.³

The original idea was to create a new TLD (top-level domain) called dot kids (.kids) but there was some controversy over the US throwing its weight around to impose US law to a domain that would be used internationally and also the bill’s sponsors felt it would take ICANN forever to actually deliberate on whether they would agree to create this for them. In light of this, the focus switched to an area more under US control, the .us domain.⁴

So as mentioned above, they created an act (the Dot Kids Implementation and Efficiency Act of 2002) and the kids dot us subdomain (.kids.us) was born. The law tasked the company under contract with NTIA to administer the .us domain at the time, Neustar, to manage the subdomain and the content moderation of the sites that registered.⁵

We have rules

Neustar started up the service and got a decent amount of registrants but not a lot of #content. It was pretty strict what could go on the site - no external hyperlinks (a walled garden), no “two-way and multi-user interactive services” (email, chat, etc.), nothing that would break existing laws, plus no: mature content, pornography, inappropriate language, violence, hate speech, drugs, alcohol, tobacco, ambling, weapons, and criminal activity. In order finance the administration of these rules, each registration came with a hefty price tag plus a “content approval fee”, which in turn probably stifled demand combined with the already stringent content rules.⁶ ⁷

We’re the kids of the dot US (woah)

At its peak, Kids dot US only had about 24 active websites, a far cry from what the crafters of the act that established it envisioned. The main kids.us site indexed these pages on the homepage. Most just consisted of Flash games which included hyperlinks, i.e. violated the rules, directing users to the download the required software (safest software out there at the time!). There also seemed to be a bunch of sites that had links to outside sites but they weren’t hyperlinked, so really any kid who learned copy/paste would be well on their way to the big bad internet.⁸ ²

Below are a selection of the mundane to strange screenshots I saw when browsing the archived website.

original kids.us index page

clearly part of the conspiracy

everyone knows what America's library is

they do stuff

Owlie Skywarn is actually pretty cool

lightning is dangerous, but also fun!

mummification, for kids!

did I mention the mummy people had a mummy bear mascot?

Santa truthers were there too

You don’t have to go home but you can’t stay here

please set it aside, and do not use it

Kids dot US shutdown officially in 2012 and an education advisory group set to the task of evaluating whether it was worth it to keep it going in some way. Spoiler alert: it wasn’t. ²

The limited scope of the subdomain makes it an ideal use case if someone wanted to do research on what went on there from a number of perspectives. It could be something similar to the study of another short lived domain, Yugoslavia (.yu), or something different entirely! Take a look at the archived 2005 site of the index (around the time with the peak number if sites) either in Wayback or Oldweb.today (for flash!) and enjoy the ride.

Hello World

Sun, 09 May 2021 04:00:50 -0400

Hi there! I’ve decided to break ground on a new website for myself that’s more under my control and at my own domain.

I’m moving away from my old site josephcarrano.wordpress.com and casting it into the abandoned graveyard. In recent years, I’ve been loath to create new posts not only because I’m lazy but also because I hate the thought of publishing content to URLs that could easily break.

Despite this worry, I won’t be migrating content at the moment or setting up redirects because I don’t feel like paying WordPress for the redirect service and it’s simple enough to leave it there neglected as it has been for the last few years. I’m mainly leaving it up in the interest of preventing link rot on the few posts that people found useful. If you want to check out the posts there, they should remain as long as wordpress.com remains free and functioning online. Maybe I’ll reconsider this later.

Anyway, I’m hoping to post a little bit more now. As much as once or twice a year even! And if you’re interested, you’ll find that content here.