Commit 35da16cd authored by Luc Didry's avatar Luc Didry
Browse files

Initial commit

parents
local/*
*.swp
*.swo
# INSTALLATION
**Warning!**
* You will still need to install the plugin `ep_delete_after_delay` on your etherpad instance if you want to be sure that pads that expired between two check loops can't be accessed.
## Dependencies
### Carton
Carton is a Perl dependencies manager, it will get what you need, so don't bother for Perl modules dependencies (but you can read the file `cpanfile` if you want).
```shell
sudo cpan Carton
```
Some modules that Carton will install need to be compiled. So you will need some tools:
```shell
sudo apt-get install build-essential libssl-dev
```
## Installation
After installing Carton:
```shell
cd /opt
git clone https://framagit.org/framasoft/pad_delete_after_delay.git delete_after_delay
cd delete_after_delay
carton install
sudo cp delete_after_delay.yml.template /etc/delete_after_delay.yml
sudo chmod 640 /etc/delete_after_delay.yml
sudo vi /etc/delete_after_delay.yml
```
The `dir` setting is really important: when a pad is expired, `delete_after_delay` stores a HTML copy of the pad in `$dir/deleted_pads`.
The plugin `ep_delete_after_delay` does the same thing, in the etherpad installation directory.
You can choose to set `dir` to the etherpad installation directory but then you will need to be sure that the `deleted_pads` is writable by the etherpad user AND the user you run `delete_after_delay` with (you can change it in `delete_after_delay@.service` with the `User` setting if you run it as a systemd service).
Warning! Verify that the expiration delays in `/etc/delete_after_delay.yml` and in your etherpad instance's `settings.json` are the same!
Check that `loop` setting is set to `false` in your etherpad instance's `settings.json`.
### delete\_after\_delay
You can see the options of `delete_after_delay` with:
```shell
/opt/delete_after_delay/delete_after_delay -h
```
If you want to start `delete_after_delay` as a systemd service:
```shell
sudo cp delete_after_delay@.service /etc/systemd/system/delete_after_delay@.service
# Change User in /etc/systemd/system/delete_after_delay@.service in needed
sudo systemctl daemon-reload
# To start delete_after_delay on instance foo:
sudo systemctl start delete_after_delay@foo.service
# To start at boot:
sudo systemctl enable delete_after_delay@foo.service
```
Edit `/etc/systemd/system/delete_after_delay@.service` to change the delay between check loops if you want.
This diff is collapsed.
[![](https://framagit.org/assets/favicon-075eba76312e8421991a0c1f89a89ee81678bcde72319dd3e8047e2a47cd3a42.ico)](https://framagit.org)
![English:](https://upload.wikimedia.org/wikipedia/commons/thumb/a/ae/Flag_of_the_United_Kingdom.svg/20px-Flag_of_the_United_Kingdom.svg.png) **Framasoft uses GitLab** for the development of its free softwares. Our Github repositories are only mirrors.
If you want to work with us, **fork us on [framagit.org](https://framagit.org)**. (no registration needed, you can sign in with your Github account)
![Français :](https://upload.wikimedia.org/wikipedia/commons/thumb/c/c3/Flag_of_France.svg/20px-Flag_of_France.svg.png) **Framasoft utilise GitLab** pour le développement de ses logiciels libres. Nos dépôts Github ne sont que des miroirs.
Si vous souhaitez travailler avec nous, **forkez-nous sur [framagit.org](https://framagit.org)**. (l'inscription n'est pas nécessaire, vous pouvez vous connecter avec votre compte Github)
* * *
# PAD\_DELETE\_AFTER\_DELAY
At [Framasoft](https://framasoft.org), we host a lot of heavily-used [Etherpad](http://etherpad.org) instances.
We wanted to have time-limited pads, so [Luc Didry](https://luc.frama.io) developped a [plugin](https://framagit.org/luc/ep_delete_after_delay) which deletes pads after a configured delay.
But we have so many pads (more than 30k on some instances) that the plugin fails to work correctly: it causes a really huge load on the etherpad processus at start and stuck it.
So we developped an external script to delete the expired pads without loading the etherpad processus.
`delete_after_delay` asks an etherpad instance for the list of its pads, then check (and delete if expired) all of them.
# Installation
Have a look at the [INSTALL.md file](INSTALL.md)
# License
GNU General Public License, version 3. Check the [LICENSE file](LICENSE).
requires 'Etherpad';
requires 'IO::Socket::SSL';
requires 'Config::YAML';
# carton snapshot format: version 1.0
DISTRIBUTIONS
CPAN-Meta-2.150010
pathname: D/DA/DAGOLDEN/CPAN-Meta-2.150010.tar.gz
provides:
CPAN::Meta 2.150010
CPAN::Meta::Converter 2.150010
CPAN::Meta::Feature 2.150010
CPAN::Meta::History 2.150010
CPAN::Meta::Merge 2.150010
CPAN::Meta::Prereqs 2.150010
CPAN::Meta::Spec 2.150010
CPAN::Meta::Validator 2.150010
Parse::CPAN::Meta 2.150010
requirements:
CPAN::Meta::Requirements 2.121
CPAN::Meta::YAML 0.011
Carp 0
Encode 0
Exporter 0
ExtUtils::MakeMaker 6.17
File::Spec 0.80
JSON::PP 2.27300
Scalar::Util 0
perl 5.008001
strict 0
version 0.88
warnings 0
Class-Method-Modifiers-2.12
pathname: E/ET/ETHER/Class-Method-Modifiers-2.12.tar.gz
provides:
Class::Method::Modifiers 2.12
requirements:
B 0
Carp 0
Exporter 0
ExtUtils::MakeMaker 0
base 0
perl 5.006
strict 0
warnings 0
Config-JSON-1.5202
pathname: R/RI/RIZEN/Config-JSON-1.5202.tar.gz
provides:
Config::JSON 1.5202
requirements:
ExtUtils::MakeMaker 6.30
File::Temp 0.18
JSON 2.16
List::Util 1.19
Moo 0
Test::Deep 0.095
Test::More 0.7
Config-YAML-1.42
pathname: M/MD/MDXI/Config-YAML-1.42.tar.gz
provides:
Config::YAML 1.42
requirements:
ExtUtils::MakeMaker 0
Test::More 0
YAML 0.35
Devel-GlobalDestruction-0.14
pathname: H/HA/HAARG/Devel-GlobalDestruction-0.14.tar.gz
provides:
Devel::GlobalDestruction 0.14
requirements:
ExtUtils::MakeMaker 0
Sub::Exporter::Progressive 0.001011
perl 5.006
Etherpad-1.2.13.0
pathname: L/LD/LDIDRY/Etherpad-1.2.13.0.tar.gz
provides:
Etherpad 1.002013000
requirements:
ExtUtils::MakeMaker 0
Mojolicious 6.0
IO-Socket-IP-0.38
pathname: P/PE/PEVANS/IO-Socket-IP-0.38.tar.gz
provides:
IO::Socket::IP 0.38
requirements:
IO::Socket 0
Socket 1.97
Test::More 0.88
IO-Socket-SSL-2.038
pathname: S/SU/SULLR/IO-Socket-SSL-2.038.tar.gz
provides:
IO::Socket::SSL 2.038
IO::Socket::SSL::Intercept 2.014
IO::Socket::SSL::OCSP_Cache 2.038
IO::Socket::SSL::OCSP_Resolver 2.038
IO::Socket::SSL::PublicSuffix undef
IO::Socket::SSL::SSL_Context 2.038
IO::Socket::SSL::SSL_HANDLE 2.038
IO::Socket::SSL::Session_Cache 2.038
IO::Socket::SSL::Utils 2.014
requirements:
ExtUtils::MakeMaker 0
Net::SSLeay 1.46
Scalar::Util 0
JSON-2.90
pathname: M/MA/MAKAMAKA/JSON-2.90.tar.gz
provides:
JSON 2.90
JSON::Backend::PP 2.90
JSON::Boolean 2.90
requirements:
ExtUtils::MakeMaker 0
Test::More 0
JSON-PP-2.27400
pathname: M/MA/MAKAMAKA/JSON-PP-2.27400.tar.gz
provides:
JSON::PP 2.27400
JSON::PP::Boolean 2.27400
JSON::PP::IncrParser 2.27400
requirements:
ExtUtils::MakeMaker 0
Test::More 0
Module-Build-0.4220
pathname: L/LE/LEONT/Module-Build-0.4220.tar.gz
provides:
Module::Build 0.4220
Module::Build::Base 0.4220
Module::Build::Compat 0.4220
Module::Build::Config 0.4220
Module::Build::Cookbook 0.4220
Module::Build::Dumper 0.4220
Module::Build::Notes 0.4220
Module::Build::PPMMaker 0.4220
Module::Build::Platform::Default 0.4220
Module::Build::Platform::MacOS 0.4220
Module::Build::Platform::Unix 0.4220
Module::Build::Platform::VMS 0.4220
Module::Build::Platform::VOS 0.4220
Module::Build::Platform::Windows 0.4220
Module::Build::Platform::aix 0.4220
Module::Build::Platform::cygwin 0.4220
Module::Build::Platform::darwin 0.4220
Module::Build::Platform::os2 0.4220
Module::Build::PodParser 0.4220
requirements:
CPAN::Meta 2.142060
CPAN::Meta::YAML 0.003
Cwd 0
Data::Dumper 0
ExtUtils::CBuilder 0.27
ExtUtils::Install 0
ExtUtils::Manifest 0
ExtUtils::Mkbootstrap 0
ExtUtils::ParseXS 2.21
File::Basename 0
File::Compare 0
File::Copy 0
File::Find 0
File::Path 0
File::Spec 0.82
File::Temp 0.15
Getopt::Long 0
Module::Metadata 1.000002
Parse::CPAN::Meta 1.4401
Perl::OSType 1
Pod::Man 2.17
TAP::Harness 3.29
Test::More 0.49
Text::Abbrev 0
Text::ParseWords 0
perl 5.006001
version 0.87
Module-Runtime-0.014
pathname: Z/ZE/ZEFRAM/Module-Runtime-0.014.tar.gz
provides:
Module::Runtime 0.014
requirements:
Module::Build 0
Test::More 0
perl 5.006
strict 0
warnings 0
Mojolicious-7.10
pathname: S/SR/SRI/Mojolicious-7.10.tar.gz
provides:
Mojo undef
Mojo::Asset undef
Mojo::Asset::File undef
Mojo::Asset::Memory undef
Mojo::Base undef
Mojo::ByteStream undef
Mojo::Cache undef
Mojo::Collection undef
Mojo::Content undef
Mojo::Content::MultiPart undef
Mojo::Content::Single undef
Mojo::Cookie undef
Mojo::Cookie::Request undef
Mojo::Cookie::Response undef
Mojo::DOM undef
Mojo::DOM::CSS undef
Mojo::DOM::HTML undef
Mojo::Date undef
Mojo::EventEmitter undef
Mojo::Exception undef
Mojo::Headers undef
Mojo::HelloWorld undef
Mojo::Home undef
Mojo::IOLoop undef
Mojo::IOLoop::Client undef
Mojo::IOLoop::Delay undef
Mojo::IOLoop::Server undef
Mojo::IOLoop::Stream undef
Mojo::IOLoop::Subprocess undef
Mojo::JSON undef
Mojo::JSON::Pointer undef
Mojo::Loader undef
Mojo::Log undef
Mojo::Message undef
Mojo::Message::Request undef
Mojo::Message::Response undef
Mojo::Parameters undef
Mojo::Path undef
Mojo::Reactor undef
Mojo::Reactor::EV undef
Mojo::Reactor::Poll undef
Mojo::Server undef
Mojo::Server::CGI undef
Mojo::Server::Daemon undef
Mojo::Server::Hypnotoad undef
Mojo::Server::Morbo undef
Mojo::Server::PSGI undef
Mojo::Server::PSGI::_IO undef
Mojo::Server::Prefork undef
Mojo::Template undef
Mojo::Transaction undef
Mojo::Transaction::HTTP undef
Mojo::Transaction::WebSocket undef
Mojo::URL undef
Mojo::Upload undef
Mojo::UserAgent undef
Mojo::UserAgent::CookieJar undef
Mojo::UserAgent::Proxy undef
Mojo::UserAgent::Server undef
Mojo::UserAgent::Transactor undef
Mojo::Util undef
Mojo::WebSocket undef
Mojolicious 7.10
Mojolicious::Command undef
Mojolicious::Command::cgi undef
Mojolicious::Command::cpanify undef
Mojolicious::Command::daemon undef
Mojolicious::Command::eval undef
Mojolicious::Command::generate undef
Mojolicious::Command::generate::app undef
Mojolicious::Command::generate::lite_app undef
Mojolicious::Command::generate::makefile undef
Mojolicious::Command::generate::plugin undef
Mojolicious::Command::get undef
Mojolicious::Command::inflate undef
Mojolicious::Command::prefork undef
Mojolicious::Command::psgi undef
Mojolicious::Command::routes undef
Mojolicious::Command::test undef
Mojolicious::Command::version undef
Mojolicious::Commands undef
Mojolicious::Controller undef
Mojolicious::Lite undef
Mojolicious::Plugin undef
Mojolicious::Plugin::Config undef
Mojolicious::Plugin::Config::Sandbox undef
Mojolicious::Plugin::DefaultHelpers undef
Mojolicious::Plugin::EPLRenderer undef
Mojolicious::Plugin::EPRenderer undef
Mojolicious::Plugin::HeaderCondition undef
Mojolicious::Plugin::JSONConfig undef
Mojolicious::Plugin::Mount undef
Mojolicious::Plugin::PODRenderer undef
Mojolicious::Plugin::TagHelpers undef
Mojolicious::Plugins undef
Mojolicious::Renderer undef
Mojolicious::Routes undef
Mojolicious::Routes::Match undef
Mojolicious::Routes::Pattern undef
Mojolicious::Routes::Route undef
Mojolicious::Sessions undef
Mojolicious::Static undef
Mojolicious::Types undef
Mojolicious::Validator undef
Mojolicious::Validator::Validation undef
Test::Mojo undef
ojo undef
requirements:
ExtUtils::MakeMaker 0
IO::Socket::IP 0.37
JSON::PP 2.27103
Pod::Simple 3.09
Time::Local 1.2
Moo-2.002005
pathname: H/HA/HAARG/Moo-2.002005.tar.gz
provides:
Method::Generate::Accessor undef
Method::Generate::BuildAll undef
Method::Generate::Constructor undef
Method::Generate::DemolishAll undef
Moo 2.002005
Moo::HandleMoose undef
Moo::HandleMoose::FakeConstructor undef
Moo::HandleMoose::FakeMetaClass undef
Moo::HandleMoose::_TypeMap undef
Moo::Object undef
Moo::Role 2.002005
Moo::_Utils undef
Moo::_mro undef
Moo::_strictures undef
Moo::sification undef
Sub::Defer 2.002005
Sub::Quote 2.002005
oo undef
requirements:
Class::Method::Modifiers 1.1
Devel::GlobalDestruction 0.11
Exporter 5.57
ExtUtils::MakeMaker 0
Module::Runtime 0.014
Role::Tiny 2.000004
Scalar::Util 0
perl 5.006
Net-SSLeay-1.78
pathname: M/MI/MIKEM/Net-SSLeay-1.78.tar.gz
provides:
Net::SSLeay 1.78
Net::SSLeay::Handle 0.61
requirements:
ExtUtils::MakeMaker 6.36
MIME::Base64 0
Test::More 0.60_01
perl 5.005
Role-Tiny-2.000005
pathname: H/HA/HAARG/Role-Tiny-2.000005.tar.gz
provides:
Role::Tiny 2.000005
Role::Tiny::With 2.000005
requirements:
Exporter 5.57
perl 5.006
Sub-Exporter-Progressive-0.001013
pathname: F/FR/FREW/Sub-Exporter-Progressive-0.001013.tar.gz
provides:
Sub::Exporter::Progressive 0.001013
requirements:
ExtUtils::MakeMaker 0
Test-Deep-1.123
pathname: R/RJ/RJBS/Test-Deep-1.123.tar.gz
provides:
Test::Deep 1.123
Test::Deep::All undef
Test::Deep::Any undef
Test::Deep::Array undef
Test::Deep::ArrayEach undef
Test::Deep::ArrayElementsOnly undef
Test::Deep::ArrayLength undef
Test::Deep::ArrayLengthOnly undef
Test::Deep::Blessed undef
Test::Deep::Boolean undef
Test::Deep::Cache undef
Test::Deep::Cache::Simple undef
Test::Deep::Class undef
Test::Deep::Cmp undef
Test::Deep::Code undef
Test::Deep::Hash undef
Test::Deep::HashEach undef
Test::Deep::HashElements undef
Test::Deep::HashKeys undef
Test::Deep::HashKeysOnly undef
Test::Deep::Ignore undef
Test::Deep::Isa undef
Test::Deep::ListMethods undef
Test::Deep::MM undef
Test::Deep::Methods undef
Test::Deep::NoTest undef
Test::Deep::None undef
Test::Deep::Number undef
Test::Deep::Obj undef
Test::Deep::Ref undef
Test::Deep::RefType undef
Test::Deep::Regexp undef
Test::Deep::RegexpMatches undef
Test::Deep::RegexpOnly undef
Test::Deep::RegexpRef undef
Test::Deep::RegexpRefOnly undef
Test::Deep::RegexpVersion undef
Test::Deep::ScalarRef undef
Test::Deep::ScalarRefOnly undef
Test::Deep::Set undef
Test::Deep::Shallow undef
Test::Deep::Stack undef
Test::Deep::String undef
Test::Deep::SubHash undef
Test::Deep::SubHashElements undef
Test::Deep::SubHashKeys undef
Test::Deep::SubHashKeysOnly undef
Test::Deep::SuperHash undef
Test::Deep::SuperHashElements undef
Test::Deep::SuperHashKeys undef
Test::Deep::SuperHashKeysOnly undef
requirements:
ExtUtils::MakeMaker 0
List::Util 1.09
Scalar::Util 1.09
Test::Builder 0
YAML-1.18
pathname: T/TI/TINITA/YAML-1.18.tar.gz
provides:
YAML 1.18
YAML::Any 1.18
YAML::Dumper undef
YAML::Dumper::Base undef
YAML::Error undef
YAML::Loader undef
YAML::Loader::Base undef
YAML::Marshall undef
YAML::Mo undef
YAML::Node undef
YAML::Tag undef
YAML::Type::blessed undef
YAML::Type::code undef
YAML::Type::glob undef
YAML::Type::ref undef
YAML::Type::regexp undef
YAML::Type::undef undef
YAML::Types undef
YAML::Warning undef
yaml_mapping undef
yaml_scalar undef
yaml_sequence undef
requirements:
ExtUtils::MakeMaker 0
perl 5.008001
#!/bin/bash
cd /opt/delete_after_delay/
/usr/local/bin/carton exec ./delete_after_delay.pl -- $@
#!/usr/bin/perl
# vim:set sw=4 ts=4 sts=4 ft=perl expandtab:
use Mojo::Base -strict;
use Getopt::Long;
use File::Spec;
use Config::YAML;
use Mojo::Collection 'c';
use Mojo::Util qw(slurp spurt);
use Etherpad;
use Data::Dumper;
my $c = Config::YAML->new(config => '/etc/delete_after_delay.yml');
my $instance;
my ($help, $verbose, $daemon, $dry_run, $sleep, $expired) = (0, 0, 0, 0, 600, 0);
shift @ARGV if ($ARGV[0] eq '--');
GetOptions(
'instance|i=s' => \$instance,
'verbose|v' => \$verbose,
'daemon|d' => \$daemon,
'dry-run|n' => \$dry_run,
'sleep|s=i' => \$sleep,
'help|h' => \$help,
);
print_usage(0) if $help;
print_usage(1) unless ($instance && $c->{$instance});
my $ec = Etherpad->new(
url => $c->{$instance}->{url},
apikey => $c->{$instance}->{key},
);
print_usage(2) unless $ec->check_token();
my $delay = $c->{$instance}->{del};
print_usage(3) unless ($delay && $delay > 0);
my $text = $c->{$instance}->{text} || 'The content of this pad has been deleted since it was older than the configured delay.';
my $dir = File::Spec->catdir($c->{$instance}->{dir}, 'deleted_pads');
mkdir $dir unless (-e $dir);
print_usage(4) unless (-d $dir && -w $dir);
if ($daemon) {
while (1) {
say 'Launching deletion loop' if $verbose;
delete_after_delay();
sleep $sleep;
}
} else {
say 'Launching deletion loop' if $verbose;
delete_after_delay();
}
sub delete_after_delay {
say 'Getting pads list' if $verbose;
my $pads = c($ec->list_all_pads());
$pads->each(
sub {
my ($e, $num) = @_;
say 'Checking pad '.$e if $verbose;
my $last_edit = $ec->get_last_edited($e);
my $revs = $ec->get_revisions_count($e);
if ($last_edit && $revs) {
my $time = time * 1000;
if (($time - $last_edit) > ($delay * 1000)) {
say 'Will delete '.$e.' (time diff: '.($time - $last_edit).')' if $verbose;
say sprintf 'Expired pad: %s', ($e) if (!$verbose && $dry_run);
$expired++;
unless ($dry_run) {
say 'Getting HTML version of '.$e if $verbose;
my $html = $ec->get_html($e);
if ($html) {
my $file = File::Spec->catfile($c->{$instance}->{dir}, 'deleted_pads', $e.'-'.$time.'.html');
say 'Putting HTML version of '.$e.' in '.$file if $verbose;
spurt $html, $file;
say 'Deleting '.$e if $verbose;
if ($ec->delete_pad($e)) {
say 'Recreating '.$e if $verbose;
$ec->create_pad($e, $text);
}
}
} else {
say 'Not deleting '.$e.' since dry-running' if $verbose;
}
}
}