Skip to content

Add to bibliography update a check of every URL if broken vs. accessible. #337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
MattHeffron wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Add to bibliography update a check of every URL if broken vs. accessible. #337

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
abb2809
Initial attempt. Too many "false positive" indicating broken when not.
MattHeffron Jun 18, 2026
ff0fcd4
Added output of curl errors and headers from bibSplit.pl, for determi…
MattHeffron Jun 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions scripts/bibSplit.pl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,20 @@ sub sanitize_text {
my $itemTitle = sanitize_text($obj->{title} // '');
my $title = $itemTitle eq '' ? "title: ''" : "title: |\n $itemTitle\n";

my $urlSource = defined $obj->{url} ? $obj->{url} : '';

print STDERR qq{$key --> "$urlSource" on "$itemTitle".\n};
# test if the URL is accessible
if ($urlSource ne '') {
if (system(qq{curl --output /dev/null --silent --show-error --head --fail --dump-header "$key.hdr" --location --referer "https://interlisp.org/history/bibliography;auto" "$urlSource"}) != 0) {
print STDERR qq{URL is broken or unreachable.\n};
}
else
{
unlink("$key.hdr");
}
}

# Abstracts can be multi-line and contain multiple paragraphs. Place YAML keyword on
# one line and follow it with the abstract indented on subsequent lines.
my $abs = sanitize_text($obj->{abstract} // '');
Expand Down Expand Up @@ -101,8 +115,6 @@ sub sanitize_text {
$itemEditors =~ s/\n$//u; # strip trailing newline
}

my $urlSource = defined $obj->{url} ? $obj->{url} : '';

# Modified date
my $dateModified = defined $obj->{dateModified} ? $obj->{dateModified} : '';

Expand Down Expand Up @@ -165,7 +177,7 @@ sub sanitize_text {
} elsif ($type eq 'entry-encyclopedia') {
$extraFields = "encyclopedia_title: $encyclopediaTitle\n";
} else {
print STDERR "Warning: unhandled type \"$type\" for key \"$key\"\n";
# print STDERR "Warning: unhandled type \"$type\" for key \"$key\"\n";
}

# Todo: Remove writing the json file once we're happy with the markdown files
Expand Down