1. touch -d “Jun 01 00:00 2011″ /tmp/.date1
2. enter into your BIG dir
3. press C-x ! (External panelize)
4. add new command like a “find . -type f \( -newer /tmp/.date1 \) -print”

I’ve used a slightly different approach, specifying desired date right in the command line of External Panelize:

1. enter your directory with many files
2. press C-x ! (External Panelize)
3. add a command like find . -type f -newermt "2017-02-01 23:55:00" -print (man find for more details)

In both cases, the created panel will only have files matching your search condition.

This will duplicate both metadata and btrfs system data.
Verify with: sudo btrfs fi df /toplevel:

Data, single: total=10.00GiB, used=3.88GiB
System, DUP: total=64.00MiB, used=4.00KiB
Metadata, DUP: total=512.00MiB, used=286.18MiB
GlobalReserve, single: total=96.00MiB, used=0.00B

Explanation: on SSDs, mkfs.btrfs creates metadata in single mode (because of widely spread SSD deduplication algorithms negating duplicate entries). However, second copy of metadata increases recovery chances, especially so if your SSD does not deduplicate writes. Hence the desire to add metadata/systemdata duplication after the filesystem is created.

My current email server was configured eons ago, it works well,
but I have no desire to painfully transfer all the configuration…
Better install something new, shiny and exciting, right?

I had 3 #self-hosted, #mail-server bookmarks:

• Mail-in-a-box
• iRedMail
• Modoboa
• Sovereign

(Sovereign, the 4th one, was addded after reading more about Mail-in-a-box.)

Here are my notes on what seemed important about these 4.

• iRedMail
• has free and paid web-UIs
• no DNSSEC, DMARC, HSTS
• amavisd with clamav
• has useful manual parts
• containerized
• not attractive
• Modoboa
• less sophisticated than Sovereign or Mail-in-a-box
• web-UI, also for amavisd filters
• overall: focuses on better UI
• has useful manual parts
• recent (experimental?) LetsEncrypt support
• has (some) unit tests
• containerized
• not that attractive
• Sovereign
• has more than I need, but components can be deactivated
• has EncFS support (useful, but questionable because of reboots…)
• no dedicated web-interface, configs are text
• has proper testing against a vagrant virtual machine
• can be dockerized using github.com/kisamoto/dancible
• attractive as “the next solution”, or to borrow EncFS support
• Mail-in-a-box
• the most sophisticated email server (except for EncFS which is not used here)
• simple but useful web-UI
• no amavisd, clamav, UI for filters
• has good relaying manual
• more or less requires a separate machine (overwrites configs?)
• has no well-established testing, not even for development; this is being worked on as of New Year 2017
• problems with owncloud (which I don’t really need)
• hub.docker.com/r/mtrnord/mailinabox/ , github.com/mail-in-a-box/mailinabox/blob/docker/containers/docker/run
• postscreen is not yet configured, it is not obvious if it were beneficial
• the most attractive; might be reasonable to fork and modify (e.g. drop owncloud?)

MIAB appeared really attractive,
but then – do I really want to dedicate one of the VPS to the mail server only?
Not in my case – too low emails volume/traffic.

So running it in an LXC (or some other) container would make sense.
And this is actually possible, some of the users over at MIAB’s discussion forum
have been running MIAB inside docker container for over a year now with no issues.
(An extra upside is that web-UI can be left unexposed, preventing external access to it.)
A possible long-term downside is, of course, lack of tests – Sovereign looks much better in this regard.

Sovereign looks very good overall. In fact, MIAB feels like
“Sovereign’s email component + webui for it” (MIAB was inspired by Sovereign).

One extra MIAB-specific feature is DNSSEC support.
MIAB takes on the role of your nameserver, and thus is able to setup (and refresh, when necessary)
all the DKIM/DNSSEC/etc-relevant DNS records for you.

As soon as I’ve started adding “containerization” to the mix, dozens of other projects entered my field of view:

• github.com/indiehosters/email, inspired by MIAB, looks ok; lacks webmail, fail2ban, SPF, DANE, DNSSEC, but uses vimbadmin instead of a custom-coded MIAB UI
• github.com/tomav/docker-mailserver looks great! No UI, no SQL backend, only 2 text files (accounts and aliases) for all configuration – yay!
• github.com/lava/dockermail, much less active/polished, not really interesting
• github.com/frankh/docker-compose-mailbox adds roundcube and vimbadmin containers; uses SQL; not sure why it has only 10 stars on github…
• github.com/adaline/dockermail – looks ok, less active and seems simpler than docker-mailserver
• poste.io : has free (downloadable) and 2 paid versions; packed with many features and containerized; there is no Dockerfile, but of course you can examine what’s inside the public image anyway; actually, looks good – not sure how posteio-specific the data directory structure is, though… still something to try
• mailgun.com – SMTP service with a more than sufficient free quota for a few low-traffic websites; can be coupled with some forwarding service to avoid any need in an email server; but not this time, I want a mail-server
• yunohost.org : I’m not entirely sure why this is here, maybe it does have email support built-in? ok, yes it does – this is a debian-based “home-server” software, which also includes LDAP and SSO and XMPP and DNS and nginx. Hmm, not bad. I wonder how well it works out of the box.
• kolab.org : groupware; looks interesting as well, but I have no group (yet) to have a use for a full groupware solution
• not reviewed: mailcow.email, mailcow.email/dockerized, github.com/andryyy/mailcow

Finally, one can build an own LXC container, either by following this ArsTechnica series,
or after examining the install scripts of MIAB or Sovereign.
Then automate all of this, keep it well-maintained – and there you have it, one more mail-server solution!

To re-cap:

• MIAB looks very good – feature-rich, easy to install, and just works – you should try it!
• docker-mailserver looks great – I should try it!
• poste.io, yunohost.org and kolab.org are also some interesting solutions to try, along with Sovereign

Not much of a summary, but this is definitely an accurate reflection of reality.

But I always keep looking for new/improved tools, as right now I feel the best one does not exist…
(If the best one can exist at all – requirements and conditions change all the time, so there is no fixed perfect immovable target.)

I have been contemplating trying out the TSW methodology, but neither Keep nor Trello are quite there yet.
I ended up using Evernote; after recent management changes and actually trying to become profitable it may as well last long enough.

Everything was fine and calm until I have found workflowy yesterday.
In essence, it is very similar to the text-file-based system that I have been using for at least half a year.

Briefly, it is a web-based text editor on steroids, with possibly infinite nesting lists and seemingly full keyboard shortcuts control – no mouse needed.
I recommend that you try the demo – it seems to be fully functional, and there is no need to sign up.

This discovery made me read through pages and pages of this class of software tools.
Here is a very brief summary of my findings:

• org-mode for emacs: taking notes, organizing to-do lists and projects, writing structured text; I can definitely see the benefits – especially for tasks/projects management; however, for rich content – with attached/embedded files/images – this probably won’t work that well; see also orgzly;
• TagSpaces (also on github) is a tags-based files manager; cross-platform, offline, stores tags in filenames (in square brackets) – this is probably the least interfering/locking-in solution, the only things changing are file names; TagSpaces users are expected to synchronize the tag-controlled filesystem (which can be some specific directory tree) using 3rd-party tools such as [BT]Sync, SyncThing, Dropbox, Box, etc;
• many kinds of Evernote-like tools and wikis, that people also use to take notes; the two most mature and usable and feature-rich are probably Laverna (looks great, uses markdown, can be self-hosted, unclear how to search by multiple tags) and Paperwork (same thing with multiple tags); WizNote also deserves a mention; among other tools: git-backed magpie, yipgo, Marks, KeepNote, zim, etc;
• hamster-GTD is a (yet another?) variation of TSW/GTD – not a tool, but a system;
• Artificer (formerly Overlord S-RAMP), a system for any kind of interconnected, hierarchical data; see also Artificer GTD example;
• magnolial, workflowy clone (haven’t tried it yet); another clone is HackFlowy – trying its offline demo shows that only a basic list functionality is present.

Now that I think of it, TagSpaces is a neat idea…
Especially for photos – assigning tags actually updates filenames, which is great for photos.
And you can easily search by those tags later in TagSpaces.
In fact, TagSpaces looks very interesting for organizing lots of directory/file-based data.

Laverna looks quite exciting! And seems to be actively developed.
But there seems to be nothing quite comparable to WorkFlowy… Need to test it some more.

the notebook you are searching in has been moved or renamed since the saved search was created

(which is not true).

I had this problem, and found a solution.

Go to your Evernote on a client where you can edit saved searches (Windows for me),
edit all the searches, and make sure that notebook name is quoted in the search (and also, possibly, with all proper letter cases).

I found this solution by first creating a search from the web-beta interface, it looked like this: notebook:"Mynotebook" tag:1-now
All the crossed-out searches (despite working totally fine on Windows) looked like this: notebook:Mynotebook tag:1-now
or even like this (note the lower-case 1st letter of the notebook name): notebook:mynotebook tag:1-now.

After editing saved searches and synchronizing, they all appear (and work) just fine in the beta web-interface.

If you cannot edit your searches right now, there is another workaround: all the saved searches work fine for me from the Shortcuts menu (a star in the left panel).

Hope this helps!

Since 2014 I am participating in a local 5km mass-run (with over 15k participants). My time is between nothing special and fairly good. I will run this year as well.

Previously, I did not train systematically. I would start only when it was comfortably warm (mid-late April), running 4-6km 1-3 times a week.
I would also stop training very soon after the run, and definitely stop if it was getting colder (early October).
After every winter, it actually felt like I am starting my training from scratch.

Eventually, I have adopted an 18-session training regimen, which I found to be very easy for me, and also very efficient.
I had also tried to keep training for longer after the run (which is in summer).

This year, thanks to highly-enthusiastic co-workers, we started light jogging already in January.
One of my co-workers has a nearly-full set of gadgets, including a heart-rate monitor, and was keeping his pulse under a certain threshold.
I have only used a very basic Xiaomi Mi Band fitness/sleep tracker before, and got interested in a more quantified self.
After some reading, I have decided to buy a heart-rate monitor.

Here are my requirements for a heart-rate monitor:

• should measure heart-rate well; based on some reading, I’d prefer to have a chest strap: optical pulse measuring is a tad less precise, it may work worse with hairy (man’s) hands, and (for now) I only plan measuring my heart rate when I’m actually running/bicycling, not all day long;
• should be convenient to use – a wrist display is fine, but carrying a smartphone during a run isn’t;
• should not be expensive: I do not know yet how useful it will be to me, so I do not want to spend more than 150 EUR on it, so the likes of Polar V800 are, unfortunately, outside the scope of my comparison…

I have come across four popular models: Beurer PM 25 (30 EUR), Sigma PC 25.10 (40 EUR), Polar FT60F (70 EUR), and Polar M400 (150 EUR).
These devices differ quite significantly in their class/functionality/price (although all are relatively affordable), and with this blog post I’ll be trying to figure out which one makes the most sense for me – and for any other casual/amateur jogger/runner.

Both Beurer PM 25 and Sigma PC 25.10 (does that 25 mean the same in both models? no idea…) come with a chest strap included, and they both are simple heart-rate monitor watches: you start them before the run, they show your pulse during the run, and some statistics after the run.
For the next run, you need to either reset the data, or the new run will influence your averages from the previous run – so, effectively, both have memory for 1 run.
Both use replaceable CR2032 batteries – one for the watch and one more for the strap, and it should last many months before requiring replacement.
There are no Bluetooth/cable connectivity options.
Both are actual watches (show time/date).
See a table below for a brief comparison of features.
Oh, they also look different – see above

Beurer PM 25 vs Sigma PC 25.10: simple heart-rate watches

Of these two, I’d probably go with Sigma, mostly because of less interference-sensitive coded radio link, and also some rudimentary training history functionality.
Going from Sigma to Polar FT60F does not feel like a major upgrade: for almost double the price, you get

• individual training program,
• HR-based fitness test, and
• memory for 50 runs instead of just 1.

You also get the ability to use additional GPS and running sensors with the watch, but FT60F only has rudimentary support for those sensors – so no rich data.
Also, it appears that getting data off the watch will need yet another additional device – it is getting complicated enough to still prefer the Sigma model over FT60F.

Strictly speaking, PC 25.10 satisfies all the requirements for an HR monitor that I’ve formulated above.
But the GPS-enabled M400 is much more advanced (attractive?), and still has an acceptable price.

Polar FT60F vs Polar M400

Polar M400 costs 150 EUR (almost 4 times the Sigma model price), but comes with an H7 chest strap (which is sold separately for 60+ EUR), and is the most feature-rich gadget in this round-up.
The most significant additions are a built-in GPS module and an easy data sync with Polar’s phone app and online service.

GPS allows figuring out where you had actually run when you got lost, which did happen a few times to our running group
In fact, M400 has a feature to backtrack your route back to the starting point!
(Although it will likely simply tell you a straight-line azimuth and maybe distance to the starting point…)
GPS feature could also be useful for bicycle days, for tracking routes taken, although 6-8 hours of GPS+HR operation might become a limiting factor here.
This can be augmented with a battery pack, to recharge during breaks.
Which, in turn, can be harmful for the device, as charging it while the USB port is damp causes quick corrosion of the said port.
GPS also tracks your speed and distance, so you can see how your speed improves while HR stays the same.

Easy sync enables retrospective performance analysis and nice visualizations of your training achievements.
Using the Flow, Polar’s online service, you can also configure multiple types of activity and upload up to 20 of them to the watch.
This makes M400 a fairly versatile tracking tool.

Speaking of daily activity tracking, it is probably nice, but absence of a smart alarm significantly undermines the activity tracking aspect of M400.

Finally, recent firmware update to M400 added some smartwatch functionality, like messages and notifications being sent to the watch, and being able to mute or pick up incoming calls – but that’s it.

So, Sigma PC 25.10 or Polar M400?…
This was a very hard decision to make.
I like the M400 a lot – this is a great sports watch!
But it’s daily activity tracking feels like a gimmick – I will not wear a chest strap all day long,
and without it my old Mi Band does the same tracking (plus has a “smart” vibration alarm).
I would love to have a heart-rate-enabled activity tracker – especially if it can also double as a precise running HR monitor.
But M400 is not quite there yet, and it’s great GPS and sync/history features are not strictly a necessity for any of my activities –
I just want to know if I’m running/bicycling hard enough, but not too hard

So I have decided to go with a simpler Sigma this time…
But I’ll also keep tracking the brand-new vivoactive HR, coming to markets in April;
based on some previews (and also vivoactive reviews), it is going to be a great all-round (smart-)watch/activity-tracker/sports-watch with a built-in optical HRM.

Feel free to share your experiences and thoughts on the subject.

Ah, and the promised 18 easy training sessions:

Easy 18-session 30-minute / 5 km run training program

1. 1 minute running, 1 minute walking; total 20 minutes (this includes the last minute of walking, the same below)
2. 1m running, 1m walking, 2m running; 1m walking, total 25m
3. 2m running, 1m walking; total 30m
4. 5x (3 minutes running, 1 minute walking); total 20 minutes
5. 4x (3 running, 1 walking, 2 running, 1 walking); total 28 minutes
6. 5x (4 running, 1 walking); total 25 minutes
7. 8x (3 running, 1 walking); total 32 minutes
8. 5x (5 running, 1 walking); total 30 minutes
9. 2x (5 running, 1 walking, 8 running, 1 walking); total 30
10. 3x (8 running, 1 waking); total 27
11. 8m run, 1m walk, 10m run, 1m walk, 8m run, 1m walk (total 29)
12. 3x (10 running, 1 walking); total 33
13. 10m running, 1m walking, 15m running, 1m walking, 10m running, 1m walking; total 38
14. 5m running, 1m walking, 20m running, 1m walking, 10m running, 1m walking; total 38
15. 10m running, 1m walking, 20m running, 1m walking; total 32
16. 5m running, 1m walking, 25m running, 1m walking, 5m running, 1m walking; total 38
17. 10m running, 1m walking, 20m running, 1m walking, 10m running, 1m walking; total 43
18. 30 minutes running completes this preparatory course
19. add at least some more (5+) timed 5km runs

There is an easy solution to get Notes-only without editing the PDF.

If you have a PDf printer installed (most Linux distributions, and Windows 10), just do File -> Print from Impress,
then under the Print sub-header choose Notes from the Document drop-down (see picture).
Make sure to set the proper paper format for the PDF printer (A4 in my case).
Then print, and save the resulting PDF.

Sources:

• question on ask.libreoffice.org
• LibreOffice bug report

You can create hard links and symbolic links using C-x l and C-x s keyboard shortcuts. However, these two shortcuts invoke two completely different dialogs.

While for C-x s you get 2 pre-populated fields (path to the existing file, and path to the link – which is pre-populated with your opposite file panel path plus the name of the file under cursor; simply try it to see what I mean), for C-x l you only get 1 empty field: path of the hard link to create for a file under cursor. Symlink’s behaviour would be much more convenient…

Fortunately, a good man called Wiseman1024 created a feature request in the MC’s bug tracker 6 years ago. Not only had he done so, but he had also uploaded a sample mc user menu script (local copy), which works wonderfully! You can select multiple files, then F2 l (lower-case L), and hard-links to your selected files (or a file under cursor) will be created in the opposite file panel. Great, thank you Wiseman1024!

Word of warning: you must know what hard links are and what their limitations are before using this menu script. You also must check and understand the user menu code before adding it to your mc (by F9 C m u, and then pasting the script from the file).

Word of hope: 4 years ago Wiseman’s feature request was assigned to Future Releases version, so a more convenient C-x l will (sooner or later) become the part of mc. Hopefully.

More details about these tools and runlevels (in German).

• Google Keep, an awesome note-taking and to-do lists application with a really good web-interface, and free;
• Trello, convenient lists/projects/tasks management platform (especially for group work), and free;
• Google Calendar, the de facto calendar standard for Android phones, and free;
• my A5 format weekly paper planner, and… the only not free component.

It is easy to see that I am using too many tools.

In an effort to use less tools, and also to try some of the features of TSW, I’ve performed a brief search for GTD/TSW-compatible Android apps.

TSW website is built around the Evernote app. However, I am not sure if this would be a good solution for me, as I have been already using Evernote since several years for longer-term note-keeping, and thus already have a bunch of notepads, notes, and tags there. Moreover, Evernote’s website mentions something about “offline notes” in the Premium (non-free) tier for mobile apps; this hints at the requirement to have internet connectivity to be able to work with TSW+Evernote efficiently through the day.

Oh, before I forget: all the 4 tools that I am using have their purpose, with overlap between Keep and Trello.
My A5 format paper planner (weekview compact 2015) is not a simple weekly planner; it has a structure that stimulates goal-oriented planning.
More specifically, it provides means to plan:

• the entire life, by specifying (succinct) goals in several categories (personal, work, family, social, and some others);
• the next several years (there is enough space for just a few keywords for each year);
• the entire current year (as an overview or a list of goals, without too many details);
• each quarter of the current year (with more details: goals/tasks can have specific days or date ranges assigned, and have 3 priorities);
• each week has 3 priorities for what you would like to accomplish;
• there are also other important, useful, and well-designed elements, all with high attention to details.

I mostly use the paper planner for quarter-level goals and tasks.

Trello is my primary project and task management tool, both for work and personal matters (using different boards).
It also really simplifies my weekly reports: I only have to check the Done list of the primary/project work board,
and show it to my supervisor – which (showing/sharing) is also easy with Trello.

I’m using Calendar for all the events which have specific dates/times, like meetings, deadlines, celebrations, etc.

Finally, I’ve started using Keep not that long ago as a to-do list and note-taking application. It is extremely easy and quick to use, which explains this new adoption. I use it mostly as a short-term buffer for quick (shorter than 2 hours) tasks. I have 3 separate lists: home, work, and shopping The only component which is missing if I want to use TSW is tagging of individual checklist items, together with tags search. Other than that, Google Keep is plain perfect.

The first app I had a look at was DGT GTD (alpha):

• has “@-contexts”;
• has tags;
• has search for arbitrary tag combinations (both AND and OR logic);
• no web interface, uses toodledo/dropbox/ftp for sync;
• web-interface might be available through toodledo (which has its own limitations, see below);
• overall: alpha, no easy-to-use web-interface, unclear future… though otherwise seems good.

Next, I had a look at Toodledo:

• has a free, but (seemingly quite strongly) limited version; in addition, it felt
• somehow not easy to register, thus I have not tried it.

Next was MyLifeOrganized:

• way too commercial all over – you seem to need many pieces of (paid) software, (paid) cloud sync, (paid) plans…
• no web-interface and no Linux support, only Win/Mac/iOS/Android, thus have not tried this one as well.

I was leaving better contestants (like RTM, Remember The Milk) for later:

• nice, convenient, light, keyboard-friendly web-interface;
• free version syncs with web, but only once every 24 hours;
• has locations (GPS-based) and tags;
• has inbox, personal, work, study, sent pre-defined lists of tasks; you can define your own (and delete 3 of the pre-defined, if you wish);
• can search for multiple tags using brackets, logical operators, and multiple per-task attribute filters (like timeEstimate, dueDate, etc – many of these!);
• can save searches as smart lists (at least in the Android app);
• tasks cannot be ordered manually, they can only be sorted by priority, due date, or name.

I can see myself using RTM, which feels like a quality tasks-management environment. Syncing once every 24 hours is the only free version limitation that I am sensitive to, because I tend to use web-versions (Trello, Keep) while at the computer. If you are using for planning your phone only, then RTM might be a very good fit for you.

Another detail which I find inconvenient is the inability to manually sort tasks. As I know from using Keep, manually sorting smaller tasks into their logical order by dragging is quick and easy. This RTM drawback could be worked around by sorting task list by name, and devoting the first 2 characters of the task text to its number (e.g. ’06 start scaffolding’). I am still unsure about RTM.

Given the failure of the new contestants to fit my needs, I also had a quick formal look at the tools I am already using.

Evernote:

• recommended by TSW website;
• has tags and saved tag searches;
• not sure if it keeps all notes available offline – it likely needs connection to function; it may keep the most recent notes offline, though – still have to test this;
• not exactly a to-do list, thus (much more?) cumbersome to use than Google Keep (again, this wasn’t tested yet – consider this a prejudice);
• free version has a 60 MB/month data upload limit, which should be more than enough for tasks management.

I am going to try Evernote with TSW, and see if that works good enough. I’ll update the post with the results.

Google Keep:

• very easy and convenient to use checklists;
• keeps all tasks local and always available; works offline, syncs when you have connection;
• has an efficient, quick-to-use web-interface;
• does not have tagging (only colors for notes);
• lightweight in terms of size and resources needed.

I’ll keep using keep, even if some other app becomes my primary for tasks management. It is simply too good not to use.

Trello:

• has a fairly convenient web-interface (though more complicated than Keep because of more features);
• allows easy collaboration;
• supports multiple boards, containing task lists, containing tasks, containing checklists and other elements, which all together enable fairly complex project management;
• phone app caches tasks/boards that you access while online, and can later show those while offline, but
• phone app does not allow changes while offline – you must have connection for the changes to have effect;
• has tags (labels), but these are board-specific, so it is impossible to get a flat list of all tasks from all boards filtered by some labels/criteria.

Right now I have tons of tasks in Trello, so I am not going to abandon it any time soon (unless I find a perfect alternative solution). I have already seen recipes online to adapt Trello to TSW/GTD use. This will not fix the necessity for internet connection for the app to work, though. It is also quite possible that tags + global flat list of all tasks from all boards might get introduced as new features into Trello, as it is developing dynamically and new features do get added quite often… Maybe I should leave a feature request for the developers, together with a thank-you for their excellent product.

That’s it for now, I’ll update after some more app testing.

Update 1: Trello can be quite convenient as a general GTD-like (but not quite TSW-like) app. I’ve set up a separate GTD board, with lists inbox, now, next, later, 8 project/watching/reading lists, 2 goal lists (one for the current year, one with general goals), some day, contemplate (no clear decision if an item has to be done at all), waiting, done, and discarded (something from any of the other lists which is [no longer] worth doing).

Update 2: Google Keep now has labels (aka tags)! (It now also has recurring reminders, which is cool as well.) Tags seemed to be the only thing keeping (no pun intended) Keep from being a perfectly simple and lightweight GTD/TSW app! Or so I thought. Unfortunately, there does not appear to be any way to search by several labels right now. You can search by note colors, and can select a single label to list all notes that have it, but no multiple labels… One last step missing to perfection?

Leave comments if some of my statements seem wrong, or if you know a solution which would be capable of satisfying my needs

DEFAULT -d sat -H -f -p -t -W 0,40,45 -n standby -S on -m example@example.com
# Attributes 1, 230, and 231 are very important (-r 1! -r 230! -R 230! -r 231! -R 231!), but likely covered by -t.
/dev/sda -s (S/../../6/01|L/../(01|02|03|04|05|06|07)/7/00) -C 0 -I 189 -I 194
# -a implies -f and -p (through -t)
DEFAULT -d sat -a -I 194 -W 0,40,45 -n standby -o on -S on -m example@example.com
/dev/sdb -s (S/../../6/02|L/../(01|02|03|04|05|06|07)/7/02)
# This drive does not decrement Offline_Uncorrectable (198) after re-allocation,
# so only monitoring for increase, not for non-zero value.
/dev/sdc -s (S/../../6/03|L/../(01|02|03|04|05|06|07)/7/04) -U 198+
# This drive has 40 “normally”.
/dev/sdd -s (S/../../6/04|L/../(01|02|03|04|05|06|07)/7/06) -W 0,42,45

Note: explanations below are intentionally simplified; please consult man smartd.conf for more precise, complete, and up-to-date information.

Ok, so what do these settings mean, and how is this different from default settings?

By default, smartd assumes a DEVICESCAN directive, which auto-detects all HDDs, and enables reasonable default monitoring of SMART attributes.
However, there are several benefits to individually specifying your disks:

• less verbose smartd startup log messages (no messages about auto-detection and missing attributes)
• ability to run scheduled offline, short, and long SMART self-tests
• ability to monitor or exclude attributes individually for each drive (including temperature)

Reasonable default mentioned above is the -a option, equivalent to the following individual options:

-H -f -t -l error -l selftest -C 197 -U 198

This is important to know, because my only SSD has no attribute 197, no self-test log, no error log, and no automatic offline testing.
But I still want to start with quasi-default settings, and that is why the first configuration line includes all the options from -a, except those that my SSD does not support:

DEFAULT -d sat -H -f -p -t -W 0,40,45 -n standby -S on -m example@example.com

Here and in the -a options above,

• -H: monitor overall health status (passed/failed)
• -d sat: HDD type is SATA
• -f: check if any of the Usage attributes (those not marked as Pre-fail) are below the manufacturer-set thresholds
• -p: report changes in Pre-fail attributes (implied by -t below, so can be omitted)
• -t: same as -p (above) with -u (report changes in Usage attributes)
• -W 0,40,45: log a message if drive’s temperature goes above 40 degrees Celsius; log a critical messages if above 45
• -n standby: do not wake-up (spin-up) the HDD if it is in sleep or standby mode (in which platters do not spin)
• -S on: enable attributes auto-saving
• -m example@example.com: address (or several comma-separated addresses) to receive warnings from smartd

The DEFAULT directive is for convenience: options set by this directive apply to all the individual disk configuration lines below, until a different DEFAULT line is encountered.
Here, I had used it to separate all /dev/sda options into 2 logical groups: supported defaults, and drive-specific configuration.
SSD’s configuration is

/dev/sda -s (S/../../6/01|L/../(01|02|03|04|05|06|07)/7/00) -C 0 -I 189 -I 194

Here,

• -C 0: explicitly disable attribute 197 monitoring (which is not present in this SSD)
• -I 189 and -I 194: ignore attributes 189 and 194 (they both show temperature in this SSD)
• -s (S/../../6/01|L/../(01|02|03|04|05|06|07)/7/00): schedule for short and long tests

As for the tests…
I want short self-tests every Saturday night, between 1 AM and 5 AM (shifted by 1H for every disk).
I want long self-tests on the 1st Sunday of every month, between midnight and 8 AM (shifted by 2H for every disk).
This is exactly what is encoded in my configuration for all drives. The easiest way to figure out the format is to read the relevant section of man smartd.conf.

The remaining 3 drives are all HDDs, so I had defined a different common DEFAULT for them.
The only new option that we see is -U 198+, which instructs smartd to only report increases of the Offline_Uncorrectable (198) attribute.
This is necessary because my /dev/sdc does not decrement this attribute after sector re-allocation.

I hope you found this post helpful.

I am updating a multisite Drupal 6 installation. To the best of my knowledge, the only difference for Drupal 7 is that instead of the site_offline D6 variable the maintenance_mode variable is used in D7.

On Debian stable and later, you can sudo aptitude install drush and then just use it immediately after that.

Note: I recommend su webuser (or sudo -s followed by sudo -s -u webuser) before you run any non-testing drush commands, where webuser is the user which owns your web-exposed files (e.g. Debian’s default is, I think, www-data). I’ve seen a lot of recommendations to run drush as a super-user, but that does not make sense, and may actually cause problems with file ownership.

One last thing before we start: if your drush seems to work fine but hangs when untarring modules – check this solution.

1. Run some innocent command in drush to see if it produces any PHP warnings/errors you may want to fix before running actual update: drush @sites core-status. In my case, all the sites had the CacheRouter module for in-RAM caching with a server daemon back-end, which was not initialized properly when drush bootstrapped Drupal from the command line. In my case, the only working solution was to edit settings.php files of every site to comment out the CacheRouter configuration for the period of update. If you get no warnings/errors, proceed to the next step. Note: I was running drush from the Drupal’s root (directory which has top-level index.php and .htaccess files), but this should also work if you run from sites/ or even sites/sitename.
2. Here would be several more steps – copying your production website(s) to a dev-server (if you do not have one already), performing an update on the dev-server first to see if anything breaks and needs fixes, then migrating updated website(s) from the dev-server to production server. Drush actually has tools to simplify all of these procedures. However, the websites I was updating were not critical, and short downtime was not a problem, so I was updating live websites. Modify these steps as you see fit to make the process more reliable.
3. Backup databases of all your sites. With drush: drush @sites sql-dump --result-file --gzip. This puts backups somewhere into the home directory of your webuser. Backups are named with a human-readable timestamp. Of course, you can also create a manual Backup and Migrate backup, or use phpMyAdmin, or just mysqldump.
4. Backup your site’s files. This step might be unnecessary, as drush seems to backup modules it is upgrading. I would still recommend making a backup, e.g. with tar -acf multidrupal.tar.bz2 html, where html is the directory containing your multisite Drupal’s root index.php.
5. Put the websites into maintenance mode and clear all caches; see the D7-specific note above: drush @sites variable-set site_offline 1 ; drush @sites cache-clear all.
6. The actual update! The easiest way would probably be to drush @sites pm-update, but I haven’t tested that and used a process which I understand better, and which seems more reliable to me (if anything goes wrong). If in your drupal root you have sites/site1 and sites/site2, then run:
   
drush site1 pm-updatecode
drush @sites updatedb
drush site2 pm-updatecode
drush @sites updatedb

   The pm-updatecode command only updates files, and does not run database update. So with these commands I am first updating modules from site1, then running database update on all sites, then update modules of site2, and run database update on all sites again. Running drush @sites updatedb multiple times, even when there are no updates, should be safe. Take note of any warnings/errors reported, you will want to fix them later, for example:

   WARNING: Updating core will discard any modifications made to Drupal core files, most noteworthy among these are .htaccess and robots.txt. If you have made any modifications to these files, please back them up before updating so that you can re-create your modifications in the updated version of the file.

7. Disable maintenance mode. Cleaning the cache seems unnecessary, as updatedb command does that. drush @sites variable-set site_offline 0.
8. Finalize: re-enable anything disabled before the updates, fix warnings/errors you noted during the update.

This worked well for me, and I hope it works well for you.

Symptoms

• running drush @sites pm-update results in normal execution up to after answering ‘y[es]‘; then drush seems to hang indefinitely (haven’t waited beyond about 10 minutes, maybe it does produce an error after a long while);
• running the same command with --debug shows that drush hangs when trying to untar the downloaded module.tar.gz archive; there are no errors/warnings, it just hangs with no CPU usage;
• trying to untar any of the modules downloaded from drupal.org manually is also unsuccessful: tar -xzvf module.tar.gz seems to do nothing, it also hangs with zero CPU usage/time and no warnings/errors;
• interestingly, if I create some test.tar.gz locally, tar does happily extract that;
• finally, running strace tar -xzvf module.tar.gz shows a number of unexpected lines, such as references to NSS and libnss files (I am only showing some of the lines of strace output, including the last line):
  

  open(“/etc/nsswitch.conf”, O_RDONLY) = 4
  read(4, “# /etc/nsswitch.conf\n#\n# Example”…, 4096) = 683
  open(“/lib/x86_64-linux-gnu/libnss_nis.so.2″, O_RDONLY) = 4
  open(“/lib/x86_64-linux-gnu/libnss_files.so.2″, O_RDONLY) = 4
  open(“/etc/passwd”, O_RDONLY|O_CLOEXEC) = 4
  open(“/usr/lib/x86_64-linux-gnu/libnss_mysql.so.2″, O_RDONLY) = 4
  open(“/etc/group”, O_RDONLY|O_CLOEXEC) = 4
  open(“/etc/libnss-mysql.cfg”, O_RDONLY) = -1 EACCES (Permission denied)
  open(“/etc/libnss-mysql-root.cfg”, O_RDONLY) = -1 EACCES (Permission denied)
  futex(0x7fd0816e8c48, FUTEX_WAIT_PRIVATE, 2, NULL

Analysis
strace output provided enough information to understand the issue and generate a workaround. Briefly, we see tar querying users and groups information. On the system where this problem was identified, MySQL is used as a name-service back-end. This is why we see references to mysql libraries in the trace. Apparently, tar is trying to resolve some user/groups information, but for some reason does not get what it is asking in a timely manner, or possibly never gets it and will only fail/proceed when the request times out.

Workaround
Not a solution, but works: tar -xzv --numeric-owner -f module.tar.gz. The --numeric-owner switch asks tar to use numeric file/directory owner information as-is, without trying to resolve the name of the owner. This works. I have not checked strace for the workaround, but I expect to see no MySQL/NSS references in it with the switch.

To actually be able to use drush with this workaround, I had to edit drush.inc somewhere under /usr/share/drush/; look for ‘tar ‘ string, and add --numeric-owner where necessary. Do not forget that -f has to be just in front of the archive filename, otherwise your edits will not work.

you can easily fix this problem by going to Control Panel, clicking on Mail, then Show Profiles button.
Remove everything that’s there. Now start outlook again.

Note: removing all the mail profiles will disable your Novell Groupwise client.
If you still want to use non-Outlook email profiles, then the better solution is to manually create a new mail profile for Outlook.

Source: CNET forums.

• First comes make itself – the oldest and the most widely used software build tool. Stable and powerful. Still, even people who got used to using make, have some gripes about it. The most detailed list of gripes is probably here.
• SCons is a build tool written in Python. I guess I like that “configuration files are Python scripts” – maybe knowing Python is enough to use SCons, which makes SCons a better choice than make for me. SCons seems to have gained some support (scroll down for comments/discussion). There were some doubts about SCons performance (1, 2, and 3); not sure where SCons is at right now in that regard.
• waf, a Python-based framework for configuring, compiling and installing applications.
• pyDoIt is a Python automation tool. It seems to use Python syntax. It aims at bringing the power of build-tools to execute any kind of task, where a task describes some computation to be done (actions), and contains some extra meta-data. Based on the description alone, I’m quite intrigued! I wonder if anyone had already worked with pyDoIt and can share experiences?…
• Rake – Ruby make – is a simple build program with capabilities similar to those of make. Had seen a lot of positive feedback about this one – mostly regarding simplicity of use. Still [py]DoIt so far looks more attractive to me personally.
• Ruffus is a lightweight python module for running computational pipelines. Sounds like some good competition to [py]DoIt!
• Anduril is an open source component-based workflow framework for scientific data analysis. Sounds promising, though the latest downloadable version is over 400 MBs… It probably already contains a bunch of binaries and maybe even data and complete workflows for data analysis. Probably worth a look, but may turn out a little overweight for simple pipelining.
• snakemake is a scalable bioinformatics workflow engine. I get the feeling that Python is truly dominating the pipelines/workflows world: snakemake, as even the name suggests, is in Python, too. The front-page example is so simple and clear, that snakemake immediately pushes DoIt down from the 1st place! Awesome.
• Paver is a yet-another Python-based software project scripting tool along the lines of Make or Rake, designed to help out with repetitive tasks with the convenience of Python’s syntax. Sounds similar to DoIt. Have no idea how they actually compare to each other.

That is it for now.

What were your experiences with automating repetitive tasks and building simple pipelines?

As VihangD points out in his serverfault answer, the same can be achieved with aptitude, while also excluding dependent, automatically installed packages (which are included by the classical method). To create the list of packages, run aptitude search -F '%p' '~i!~M' > package_list. Here, -F '%p' asks aptitude to only print package names (instead of the default output, which also contains package state and description); search term ‘~i!~M’ asks for all non-automatically installed packages.

To install packages using the created list, run xargs aptitude --schedule-only install < package_list; aptitude install. The first of these two commands instructs aptitude to mark all the packages from the list as scheduled for installation. The second command actually performs the installation.

Hamish Downer suggests an alternative way of getting the initial package_list: using the deborphan utility, deborphan -a --no-show-section > package_list. This command asks deborphan to show a list of packages, which have no dependencies on them. Sounds very similar to what we did with aptitude above, but using deborphan will most likely result in a much shorter list of packages (on my system, deborphan printed 291 package names, aptitude printed 847, and dpkg printed 3650 package names). One more potentially important difference between aptitude- and deborphan-produced package lists is that aptitude only specifies package architecture when it is different from native (e.g. 'googleearth:i386' on a 64-bit system), while deborphan specifies architectures for all the packages (resulting in e.g. 'google-talkplugin:amd64' and 'googleearth-package:all' on a 64-bit system).

My personal choice was RStudio: it is good-looking, intuitive, easy-to-use, while powerful.

Next step would be using some R-equivalent of the excellent ipython’s Mathematica-like Notebook webinterface…

Redmine is also really cool, and is actually more feature-reach than what BitBucket has to offer, but maintaining it needs just a tiny bit more time and attention than I’m willing to spend these days. So, migration it is!

Redmine has issue 3647 titled “Data import/export system”; it is not resolved, but has a number of links to other resources. Like the redmine exporter at hostedredmine.com, which provides free hosted redmine service. Redmine itself has REST API, though I have no idea if it allows exporting all the data I may need. There’s also an XLS export plugin, but it has to be installed first, and I’m too lazy There’s also TaskAdapter, but they do not support BitBucket (yet?).

For the complete backup, I think of using the pure-ruby redmine project data export script. To migrate issues only, I’ll consider the redmine2bitbucket script.

P.S. Not implying anything (yet?), but my previous migration was from Trac to Redmine… At that time, Trac seemed to have less features than I wanted. And now I’m migrating back to “less features”, but with a benefit of no support required from me.

I had already mused on the powers of rsync before.

This time, a reminder to self on how to resume copying broken scp/mc/fish transfers, using rsync.

First, an assortment of example commands.
export RSYNC_RSH=ssh
rsync --partial file_to_transfer user@remotehost:/path/remote_file
rsync -av --partial --progress --inplace SRC DST
rsync --partial --progress --rsh=ssh host:/work/source.tar.bz2 .
rsync --partial --progress --rsh=ssh -r me@host.com:/datafiles/ ./

One could also try the --append option of rsync to base the transfer resumption on the sizes of the two files rather than verifying that their contents match.

Now a single command line explained in a little more details:
rsync -vrPtz -e ssh host:/remote_path/* /local_path/
Explained:
-e ssh rsync will use ssh client instead of rsh, which makes data exchange encrypted
-z compress file transfer
-t preserve time (other attributes, such as owner and permissions are also possible)
-P resume incomplete file transfer
-r recurse into subdirectories
-v verbose

To specify a port when using ssh you must add it to the ssh command.
Example: rsync --partial --progress --rsh="ssh -p 16703" user@host:path

I had personally settled on using pbzip2 for these simple reasons:

• performance scales quasi-linearly with the number of CPU cores (until one hits an I/O bottleneck);
• when archive is damaged, you are only guaranteed to loose the damaged block(s) of size 100-900 KiB – remaining information might be salvable.

Compared to pbzip2, neither gzip nor 7z (lzma) offer quasi-linear speedups proportional to the number of CPU cores.
pigz, the parallel gzip, does parallelize compression, but gzip compresses not as good as bzip2, and decompression is not parallel like in pbzip2.
7z is multi-threaded, but it tops out at using 2 CPU cores (see links below for tests).

pbzip2 is also quite a good choice for FASTQ data files: even if a few blocks get lost due to data corruption, this should not noticeably affect the entire dataset.
Specialized tools for FASTQ compression do exist (see e.g. this article, also Fastqz, fqzcomp, and samcomp project pages.) I think I liked fastqz quite a bit, but I still have to examine data recoverability in the case of archive damage. It is possible to use external parity tools which support file repair using pre-calculated recovery files – like the linux par2 utility, also for bzip2 archives and any other files in general – but adding parity file may negate the higher compression ratio benefits. Also, if there is no independent block structure of the archive, insufficient parity file may lead to the loss of the entire archive. In other words, this still has to be tested.

Now the long-promised web-links!

• gzip vs bzip2: bzip2 is ~2.5x slower to compress, ~10x slower to decompress, and produces 26% smaller file
• gzip, bzip2, lzma: bzip2 best on low-entropy data; gzip is the fastest; lzma is the slowest, and also not the best
• gzip, bzip, 7z: 7z decompresses faster than bzip2
• 7z, bzip2: 7z only uses up to 2 cores, bzip2 can use all; for the same time of compression, 7z yields smaller file – e.g. 7z fast 926, bzip2 maximum 987MB, both took 7 minutes
• 7z, gzip, compress, bzip2 (graphs): bzip2 is at least 2x faster than 7z, and up to 2x slower than gzip; bzip2 is the slowest at decompression, 7z is up to 2x better, while gzip is easily 6x faster
• bzip2, gzip: superuser answer provides a simple 5-axis ranking:
  decompression speed (fast > slow): gzip, zip > 7z > rar > bzip2
  compression speed (fast > slow): gzip, zip > bzip2 > 7z > rar
  compression ratio (better > worse): 7z > rar, bzip2 > gzip > zip
  availability (unix): gzip > bzip2 > zip > 7z > rar
  availability (windows): zip > rar > 7z > gzip, bzip2
• Gzip vs Bzip2 vs LZMA vs XZ vs LZ4 vs LZO (with nice tables): bzip2 is up to 3x slower to decompress; lzma may use up to 16x bzip2′s memory for decompression, which is 64 mb; bzip2 only needs up to 7.2 MB for compression; lzma needs up to 670 MB; fastest lzma is as fast as gzip, but produces smaller files; bzip2 compression time is nearly unaffected by -1..-9 settings; with the same running time, lzma-3 and bzip2-9 produced the same-size file
• lzop vs compress vs gzip vs bzip2 vs lzma vs lzma2/xz: xz-2 is comparable to bzip2-3 in file size, but is faster to compress and decompress; xz-0 is overall better than gzip-9
• gzip, bzip2: bzip2 is better at heterogeneous data
• gzip, bzip2, xz: bzip2 here is faster and smaller than gzip, while still slow to decompress