To examine the behavior of your Linux box disk caching under specific loads, see Linux write cache mystery (PDF).

To understand what is going on, see also The Linux Page Cache and pdflush (PDF) by the same author, Gregory Smith.

Another useful resource is OpenSUSE’s Tuning the Memory Management Subsystem, which nicely explains some of the kernel cache/memory-related configuration options.

ENSRNOG00000018677 1368832_at 25233
ENSRNOG00000002079 1369102_at 25272
ENSRNOG00000043451 25353
ENSRNOG00000001527 1388013_at 25408
ENSRNOG00000007390 1389538_at 25493

In the example above I need ’25353′, which does not have corresponding affy_probeset_id in the 2nd column.

It is clear how to do that:

This outputs a column of required IDs (EntrezGene in this example):

116720
679845
309295
364867
298220
298221
25353

However, I need these IDs as a comma-separated list, not as newline-separated list.

There are several ways to achieve the desired result (only the last pipe commands differ):

These solutions differ in efficiency and (slightly) in output. sed will read all the input into its buffer to replace newlines with other separators, so it might not be best for large files. tr might be the most efficient, but I haven’t tested that. paste will re-use delimiters, so you cannot really get comma-space “, ” separation with it.

Sources: linuxquestions 1 (explains used sed commands), linuxquestions 2, nixcraft.

It worked immediately for me, and default settings are good enough not to bother tweaking. Well, I know that because I did tweak a few to get more nerdiness, but normal people don’t need that.

SCR download page has packages for Ubuntu, Debian/i386, Xandros, RPM-based distributions, Gentoo – and as its free, you can of course just use the fsource, Luke!

At the time of writing, a package for Debian/amd64 was not available, but it is really easy to build one.
Here’s mine: skype-call-recorder-debian_0.8_amd64.deb

Fortunately, Alexey Smirnov (the author of DeaDBeeF) maintains a github repository deadbeef-debian which has simple instructions on installing DeaDBeeF in Debian (citing with minor edits):

1) download deadbeef from sourceforge (0.4.2 as of the moment of writing)
2) tar xvf deadbeef-X.Y.Z.tar.bz2
3) $ cd deadbeef-X.Y.Z/
4) $ git clone git://github.com/asv/deadbeef-debian.git debian
5) $ dpkg-buildpackage -rfakeroot -us -uc -i -I”.git”
6) (install any dependencies reported by dpkg-checkbuilddeps, wait for the build to finish)
7) $ sudo dpkg -i ../deadbeef_X.Y.Z-R_yourarch.deb
8) Enjoy DeaDBeeF!

Hint: dpkg-buildpackage accepts -jNumJobs option for multi-core/multi-CPU systems.

Debian has a number of packages enabling remote (over the network) backup: backupPC, backupninja, backup-manager, dirvish, duplicity, luckybackup, rdiff-backup, and some others.

In my setup I wanted backups to be initiated by the external backup server (as opposed to internal, production server initiated backup). Also, I was going to backup linux server to another linux box. Thus, it would be preferable to have support for rsync on both boxes and hardlinks on the backup box. I needed backup to be run regularly, I didn’t need any kind of GUI, and I would prefer to keep a bunch of older backups around (in case the most recent backup is of a compromised system, for example). Also, having an incremental backup of SQL databases was yet another requirement.

A few tools more or less matched my requirements: backupPC, duplicity, rdiff-backup, and dirvish.

backupPC, judging from its description and a few tutorial articles, is a cool piece of software. I think I might use it next time. But this time I really had no need for over-SMB backups (intended for Windows boxes), and no need for a nice administrative web-interface to backupPC. If not these extra features (which I don’t currently need) I’d go for backupPC. Here’s a guide for Ubuntu.

Duplicity claims to still be in beta, thus I haven’t given it a try. rdiff-backup seemed a bit too simple (and with no ready templates to get working regular backups faster), so I skipped that one as well.

Dirvish really caught my attention, partially thanks to a good and relatively short quick-start tutorial. That tutorial also mentions automysqlbackup – a nice, almost no-configuration tool to take regular dumps of MySQL databases. In combination, these two now regularly backup up my whole server to another linux box.

I would definitely recommend using dirvish+automysqlbackup for remote/off-site backup needs. However, other solutions might be as good or better for your use case.

Step-by-step:

1. Identify current swap UUID: grep swap /etc/fstab ; you should see a line starting with UUID=b4e6e… – note that UUID; let us assume our UUID is b4e6e5d8-8854-4a3e-8edb-f423ded31e2a
2. Identify swap partition device. If you are using Debian or Ubuntu, device might have been shown to you when running command #1 as a comment of the form
   

   # /dev/sdX4 none swap sw 0 0

   Just to be sure, run cat /proc/swaps (or sudo swapon -s), to be shown something like this:

   Filename Type Size Used Priority
/dev/sdX4 partition 2104504 0 -1

   You can also use partition managers to identify swap device, e.g. sudo fdisk -l. Let us assume our partition device is sdX4.

3. If that swap partition is in use, you need to disable it first: sudo swapoff /dev/sdX4
4. Now that you have both the UUID and the device, run
   sudo mkswap -L new_swap_label -U b4e6e5d8-8854-4a3e-8edb-f423ded31e2a /dev/sdX4
   Do not forget replacing sample values above (label, UUID and device) with your actual values.
5. Enable swap: sudo swapon /dev/sdX4

Now, if you wish so, you may proceed to edit /etc/fstab and replace UUID=…. with LABEL=your_new_swap_label, leaving all other spaces and values in that line untouched.

• Clementine music player
• DeaDBeeF (as in 0xDEADBEEF) is an audio player for GNU/Linux systems with X11 (though now it also runs in plain console without X, in FreeBSD, and in OpenSolaris) with CUE sheets support
• AudioPlayer51 is a free, open-source, cross-platform audio player written to handle FLAC files and CUE sheets (exclusively). If you don’t use FLAC files and CUE sheets, you won’t find AudioPlayer51 very useful.
• Amarok: only for WAV+CUE, doesn’t currently support FLAC+CUE; version 2.3.1 is promised to fully support CUE sheets
• Audacious (through the .cue plugin)
• QMMP (haven’t tested this one)
• XMMS through mp3cue (also untested)
• foobnix (untested, suggested in this comment)
• cueplay

Some extra links:

• split lossless multiple-track files by CUE sheets
• mp3split

The easiest way to start with screen is screenie – an interface to screen. To use it, you only need to memorize a single keyboard shortcut: C-a d (Ctrl+a, d). This shortcut detaches (hides) your current screen session, allowing to resume later – either with the help of screenie, or by issuing screen -r command.

After a while, with increased screen (and screenie) usage, you will find it inconvenient detaching every time you need a new screen session. Here comes the second keyboard shortcut: C-a c (Ctrl+a, c), which creates a new window within a current screen session, and switches to it.

Finally, you will find it tiresome detaching to then re-attach to a different screen. C-a “ (Ctrl+a, Shift+’) will display a list of screen windows for you to select and switch to.

Screenie, an excellent beginner’s interface to screen, is no longer needed after you memorize these 3 keyboard shortcuts.

Many will say that this is a bare minimum, and also there is much more to screen than what you present here, and that will be true. However, these 3 shortcuts are sufficient for simple but convenient everyday screen sessions management.

1. Back up important data from partition just in case
2. Find out size of ‘$Bad’ attribute in $Badclus using ntfsinfo -i 8 partition (partition is for example /dev/sda1). It will be the “Allocated size” value in the “Dumping attribute $DATA (0x80)” (there will be two 0x80 attributes. Only one has an “Allocated size” line). Let us write down (remember) this size as ntfs_size.
3. Use ntfstruncate partition 8 0x80 ‘$Bad’ 0 to set $Bad’s attribute length to zero.
4. Use ntfstruncate partition 8 0x80 ‘$Bad’ ntfs_size to set $Bad’s attribute length back to proper value ntfs_size which was recorded in step 2.
5. Boot into Windows and run chkdsk -f diskname. It will find errors and should fix them.

However, Debian’s ntfsprogs package does not have the ntfstruncate binary.

Here’s how you can easily build one yourself (you may need a few extra packages with build tools for that):

1. apt-get source ntfsprogs
2. cd linux-ntfs-2.0.0
3. ./configure && make extra
4. cd ntfsprogs && ./ntfstruncate

You can also install ntfstruncate system-wide, but I have no idea if that will interfere with the already-installed ntfsprogs package, so I just used this binary directly from linux-ntfs-2.0.0/ntfsprogs.

Bad clusters happen not only when cloning an NTFS partition between 2 physical hard disk drives. Once, after a brief blackout (with a turned on PC, and not on a UPS), the SMART status of my new HDD started showing 1 ‘Pending sector’. Correspondingly, all the SMART tests (performed with a smartctl utility) reported read errors. To help my HDD examine and fix the problem itself, I performed a write operation to that single unreadable sector, which restored it (and reset the ‘Pending sector’ counter back to zero). It was possible, because not the physical sector itself, but the hidden CRC field had errors after power failure; modern hard drives can detect and fix this kind of errors upon write operations (read operations only mark the sector as ‘pending’, because data read is not consistent with the stored CRC).

However, NTFS partition still had 4096 bytes in ‘bad clusters’. A little research revealed the presence of internal NTFS bad clusters counter, which can be reset following the procedures above.

The project does provide a 64-bit package for Ubuntu 8.10 “Intrepid Ibex”, but it has a dependency on python-zope-interface – which in Debian testing is a virtual package, currently provided by python-zope.interface. That single dependency made my aptitude complain all the time, so I edited the .deb-file, renaming depends/python-zope-interface to depends/python-zope.interface (following nice and simple instructions). After that – no more aptitude complaints, and Chandler works fine.

Download the modified Chandler 1.0.3 Debian testing amd64 package (md5: 239071715977bea2eb75f3bb54491a02).

Error in openPDF(vif) :
getOption(‘pdfviewer’) is ”; please use ‘options(pdfviewer=…)’

and you are tired of running this command every time:

> options(pdfviewer=”okular”)

then you should check if your system-wide Renviron file has proper PDF viewer set:

grep -i pdf /etc/R/Renviron

R_PDFLATEXCMD=${R_PDFLATEXCMD-${PDFLATEX-’/usr/bin/pdflatex’}}
R_RD4PDF=${R_RD4PDF-’times,hyper’}
## Default PDF viewer
R_PDFVIEWER=${R_PDFVIEWER-”}

It wasn’t in my case.

To set one, either edit the system-wide Renviron (e.g. by editing the R_PDFVIEWER=${R_PDFVIEWER-”} line to look like R_PDFVIEWER=${R_PDFVIEWER-’/usr/bin/xdg-open’} – this will use MIME types to open your preferred PDF viewer), or one of the per-user/per-directory Renviron files to fix this minor annoyance (format is the same, e.g. R_PDFVIEWER=okular ).

Miscellaneous tools exist to ease this task. This post was inspired by redet’s comparison of regexp helper tools – it could be sufficient to read only that, if you’re going to try the mentioned tools yourself. Otherwise, read on.

Kodos is an easy-to-use (but powerful enough) Python tool to verify regular expressions against sample user-provided strings. Available for both Linux and Windows. Generates ready-to-use Python regexp code.

redet assumes some knowledge of REs. Unlike all the other helper tools, it has the widest code generation coverage – just go to “Tools->Save command information” to generate code (Python, Perl, C, grep – just to name a few of the 57 (!) languages/tools/editors supported). At the same time, redet does not depend on specific versions of all supported RE engines – instead, redet automatically tests for features in available versions.

visual-regexp is similar to kodos in features.

txt2regex is a regex wizard written in bash, capable of generating code for 24 languages/editors/programs (by default, these are enabled: perl, php, postgres, python, sed and vim). It is really simple and convenient, though some understanding of “groups” and “character ranges” is highly recommended.

KRegExpEditor was a part of kde3-utils, and seems to be missing from current KDE 4.3. It is similar to kodos and visual-regexp in its philosophy and features.

I would recommend using one of kodos/visual-regexp/kregexpeditor to quickly test or design your regular expression, then use redet to generate appropriate source code. As an alternative (and when working in a console), just use txt2regexp to generate the code you need.

Tools I haven’t tried include regex coach and regexpviewer.

Please also see this extensive list of regular expression and pattern-matching related software.

[ 0.004000] Checking aperture…
[ 0.004000] No AGP bridge found
[ 0.004000] Node 0: aperture @ 20000000 size 32 MB
[ 0.004000] Aperture pointing to e820 RAM. Ignoring.
[ 0.004000] Your BIOS doesn’t leave a aperture memory hole
[ 0.004000] Please enable the IOMMU option in the BIOS setup
[ 0.004000] This costs you 64 MB of RAM
[ 0.004000] Mapping aperture over 65536 KB of RAM @ 20000000

and you are using AMD-based system w/o AGP video, then my advice is: just leave that as is, do not bother “improving”! Any tinkering with kernel boot options won’t do you any good, as the kernel has already done the best it could.

Just a note: all those messages at the top of the post should only happen if you have 4 or more GiBs of RAM. If you have less than that, and do have those messages – my experience might be inappropriate for your case.

Another note: my BIOS does not have any IOMMU settings (or “Memory hole remapping” settings), so I didn’t try that. You should check if your BIOS has IOMMU-related options first, just as kernel message suggests.

Read on for details.

My system’s kernel at the moment of writing is 2.6.30-1-amd64 (Debian testing), running on MSI’s 790GX-G65 with integrated Radeon HD 3300 video, with 4 GiB DDR3 RAM and a quad-core Phenom II CPU.

Following are the options you shouldn’t try on a similar system:

• passing iommu=noagp kernel boot option. If you run dmesg|grep -i agp, most likely you will see something in the lines of
  

  $ dmesg|grep -i agp
  [ 0.004000] No AGP bridge found
  [ 0.392098] PCI-DMA: Disabling AGP.
  [ 0.392142] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
  [ 0.774847] Linux agpgart interface v0.103

  which tells you that your kernel has automatically discovered the absence of AGP bridge, disabled it and is using 64MB of AGP aperture for IOMMU. On my system, adding that boot option had no effect.

• passing iommu=off kernel boot option. When I did this, there were loads of “nommu overflow” errors during boot, so I didn’t wait until GDM start and rebooted immediately. Clearly, tons of “overflow” errors won’t do any better than a harmless “This costs you 64 MB of RAM”. Also, systems with over 3 GiB RAM do need IOMMU.
• passing iommu=noaperture kernel boot option. On my system, this option has taken 1 more MiB from my available RAM (as reported by free -m or htop), and also enabled software double-buffering IOMMU method (in dmesg, a string similar to SW TIOLB). Performance-wise this is worse than AMD’s GART-based IOMMU, and you loose 1 more MiB of RAM ;), so don’t do this.
• passing iommu=noagp,noaperture for me had the same effect as passing iommu=noaperture alone, for the reason from point #1 of this list.
• I haven’t tried any of the memaper=[1|2|3|...] options, as I had no problems with the default 64 MiB.
• passing mem=4G may remove that message (I do not recollect if it did for me), but will eat a good deal more of your available RAM (might be because devices are now mapped within those 4G, and not above). So don’t do this.

I’m not aware of any other solutions to the missing 64 MiB RAM message. There were notices of using pci=nommconf boot option, but that is too much for such a simple message – unless you have problems booting.

Below I’m pasting IOMMU-relevant part of the linux-source-2.6.30/Documentation/x86/x86_64/boot-options.txt file, which I believe is the source of ultimate knowledge You should read that file whenever you feel like tinkering with boot options of your AMD64 machine.

I ***highlighted*** some fragments, to support the points from the list above.

IOMMU (input/output memory management unit)
 Currently four x86-64 PCI-DMA mapping implementations exist:
   1. : use ***no hardware/software IOMMU at all***
      (e.g. ***because you have < 3 GB memory***).
      Kernel boot message: "PCI-DMA: Disabling IOMMU"
   2. : ***AMD GART based hardware IOMMU***.
      Kernel boot message: "PCI-DMA: using GART IOMMU"
   3.  : Software IOMMU implementation. Used
      e.g. ***if there is no hardware IOMMU*** in the system and it is need because
      you have ***>3GB memory*** or told the kernel to us it (iommu=soft))
      Kernel boot message: "PCI-DMA: Using software bounce buffering
      for IO (SWIOTLB)"
   4.  : IBM Calgary hardware IOMMU. Used in IBM
      ***pSeries and xSeries servers***. This hardware IOMMU supports DMA address
      mapping with memory protection, etc.
      Kernel boot message: "PCI-DMA: Using Calgary IOMMU"
 iommu=[][,noagp][,off][,force][,noforce][,leak[=]
	[,memaper[=]][,merge][,forcesac][,fullflush][,nomerge]
	[,noaperture][,calgary]
  General iommu options:
    off                Don't initialize and use any kind of IOMMU.
    noforce            Don't force hardware IOMMU usage when it is not needed.
                       (default).
    force              Force the use of the hardware IOMMU even when it is
                       not actually needed (e.g. because < 3 GB memory).
    soft               Use software bounce buffering (SWIOTLB) (default for
                       Intel machines). This can be used to prevent the usage
                       of an available hardware IOMMU.
  iommu options only relevant to the AMD GART hardware IOMMU:
                 Set the size of the remapping area in bytes.
    allowed            Overwrite iommu off workarounds for specific chipsets.
    fullflush          Flush IOMMU on each allocation (default).
    nofullflush        Don't use IOMMU fullflush.
    leak               Turn on simple iommu leak tracing (only when
                       CONFIG_IOMMU_LEAK is on). Default number of leak pages
                       is 20.
    memaper[=]  Allocate an own aperture over RAM with size 32MB<4GB.
                       DAC is used with 32-bit PCI to push a 64-bit address in
                       two cycles. When off all DMA over >4GB is forced through
                       an IOMMU or software bounce buffering.
    nodac              Forbid DAC mode, i.e. DMA >4GB.
    panic              Always panic when IOMMU overflows.
    calgary            Use the Calgary IOMMU if it is available
  iommu options only relevant to the software bounce buffering (SWIOTLB) IOMMU
  implementation:
    swiotlb=[,force]
                Prereserve that many 128K pages for the software IO
                       bounce buffering.
    force              Force all IO through the software TLB.
  Settings for the IBM Calgary hardware IOMMU currently found in IBM
  pSeries and xSeries machines:
    calgary=[64k,128k,256k,512k,1M,2M,4M,8M]
    calgary=[translate_empty_slots]
    calgary=[disable=]
    panic              Always panic when IOMMU overflows
    64k,...,8M - Set the size of each PCI slot's translation table
    when using the Calgary IOMMU. This is the size of the translation
    table itself in main memory. The smallest table, 64k, covers an IO
    space of 32MB; the largest, 8MB table, can cover an IO space of
    4GB. Normally the kernel will make the right choice by itself.
    translate_empty_slots - Enable translation even on slots that have
    no devices attached to them, in case a device will be hotplugged
    in the future.
    disable= - Disable translation on a given PHB. For
    example, the built-in graphics adapter resides on the first bridge
    (PCI bus number 0); if translation (isolation) is enabled on this
    bridge, X servers that access the hardware directly from user
    space might stop working. Use this option if you have devices that
    are accessed from userspace directly on some PCI host bridge.

Recursively set directories only to drwx-rx-rx (755):

find . -type d -exec chmod 755 {} \;

Recursively set files only to rwx-r-r (644):

find . -type f -exec chmod 644 {} \;

Recursively remove carriage returns (^M) from the end of all *.php files:

find . -type f -name “*.php” -exec /home/user/dos2unix.sh {} \;

In all these cases, {} is replaced with the filename/directory find has found matching your parameters; \; at the end just stops exec processing.

However, the default FTP connection format string [username[:password]@]hostname has a drawback of not allowing the use of usernames with ‘@’-symbol in them – which is very common for virtual hostings.

One of the solutions is (done in your home directory):

1. if there is no .netrc file in your home directory — touch .netrc && chmod 600 .netrc
2. mcedit .netrc (or use vi, nano, or any other editor you prefer)
3. add the following line to the file (replace all-caps words with your actual credentials): machine HOSTNAME login USER@HOSTNAME password PASSWORD

Now, start MC, choose FTP connect, and enter only the hostname. You will be automatically logged in to the remote FTP.
This will also work for console ftp clients like lftp.

At least two convenient utilities are available in Debian Lenny for non-X Samba browsing.

smbtree (part of smbclient package) will list all visible workgroups, their servers, and share names of those servers – including “hidden” shares like C$, IPC$, ADMIN$, print$. Very handy and greppable!

samba-commander (smbc package) is a ncurses samba browser with “find file” functionality.

cat styles.css | tr -d "\r" > styles-nocarriage.css

will create styles-nocarriage.css with ^M’s removed.

Alternative syntax:

tr -d "\r" < styles.css > styles-nocarriage.css

Most editors have global replace features which allow to get rid of control characters using regular expressions (exact instructions are editor-specific).

For multiple files, try this:

for f
do
mv $f ${f}~ && tr -d "\r" <${f}~ >$f
rm ${f}~
done

Save this shell script as a file (e.g. dos2unix.sh), then do ./dos2unix.sh . This script accepts wildcards (e.g. ./dos2unix.sh *.php), so be careful!

(…)
it is a very useful tool for performing installations if you’ve got something like a LiveCD which supports your hardware.
(…)

Installing new Debian systems with debootstrap, and also as a debootstrap PDF (with comments).