Here is Kite’s farewell for you to read.

Kite did open-source many parts of their technology/software stack, though I didn’t check how comprehensive those parts are, and if that is anywhere near enough to fork/continue their work.
I wonder if there already exists an open-source project focusing on ML-based code completion for e.g. Python – let me know in the comments if you know one!

Kite cites two reasons for a shutdown: 1) technology not being quite there yet, and 2) failure to monetize.
Kite had up to 500k daily developers using the platform, but apparently extremely few were willing to pay for it.
If you do look at current ML code assistants, there seems to always exist at least some free tier – I wonder if that is forced by the same lackluster, non-paying developers attitude as for Kite.

Kite’s farewell had another interesting number: 18%.
That is by how much individual developer’s productivity could increase thanks to Kite’s assistance.
This isn’t bad at all; for a team of 5 largely independent developers, it’s almost one extra “affordable” developer.
Kite was striving to achieve a “10x improvement”, but at least to me the 18% improvement sounds good enough for sales.

I’m very curious to try some of these assistants out.
I can imagine them to be very helpful for relatively experienced developers when starting to work with a new library/ecosystem – for example, OpenVision Python bindings.
Even the common autocomplete can significantly simplify “onboarding” to a new library – and a more intelligent autocomplete should be able to help with boilerplate code (that you usually don’t have when you begin), as well as with some idiomatic expressions and statements.

Have you already played with some of the smarter code assistants?
What was your experience?
Please share

I am updating a multisite Drupal 6 installation. To the best of my knowledge, the only difference for Drupal 7 is that instead of the site_offline D6 variable the maintenance_mode variable is used in D7.

On Debian stable and later, you can sudo aptitude install drush and then just use it immediately after that.

Note: I recommend su webuser (or sudo -s followed by sudo -s -u webuser) before you run any non-testing drush commands, where webuser is the user which owns your web-exposed files (e.g. Debian’s default is, I think, www-data). I’ve seen a lot of recommendations to run drush as a super-user, but that does not make sense, and may actually cause problems with file ownership.

One last thing before we start: if your drush seems to work fine but hangs when untarring modules – check this solution.

1. Run some innocent command in drush to see if it produces any PHP warnings/errors you may want to fix before running actual update: drush @sites core-status. In my case, all the sites had the CacheRouter module for in-RAM caching with a server daemon back-end, which was not initialized properly when drush bootstrapped Drupal from the command line. In my case, the only working solution was to edit settings.php files of every site to comment out the CacheRouter configuration for the period of update. If you get no warnings/errors, proceed to the next step. Note: I was running drush from the Drupal’s root (directory which has top-level index.php and .htaccess files), but this should also work if you run from sites/ or even sites/sitename.
2. Here would be several more steps – copying your production website(s) to a dev-server (if you do not have one already), performing an update on the dev-server first to see if anything breaks and needs fixes, then migrating updated website(s) from the dev-server to production server. Drush actually has tools to simplify all of these procedures. However, the websites I was updating were not critical, and short downtime was not a problem, so I was updating live websites. Modify these steps as you see fit to make the process more reliable.
3. Backup databases of all your sites. With drush: drush @sites sql-dump --result-file --gzip. This puts backups somewhere into the home directory of your webuser. Backups are named with a human-readable timestamp. Of course, you can also create a manual Backup and Migrate backup, or use phpMyAdmin, or just mysqldump.
4. Backup your site’s files. This step might be unnecessary, as drush seems to backup modules it is upgrading. I would still recommend making a backup, e.g. with tar -acf multidrupal.tar.bz2 html, where html is the directory containing your multisite Drupal’s root index.php.
5. Put the websites into maintenance mode and clear all caches; see the D7-specific note above: drush @sites variable-set site_offline 1 ; drush @sites cache-clear all.
6. The actual update! The easiest way would probably be to drush @sites pm-update, but I haven’t tested that and used a process which I understand better, and which seems more reliable to me (if anything goes wrong). If in your drupal root you have sites/site1 and sites/site2, then run:
   
drush site1 pm-updatecode
drush @sites updatedb
drush site2 pm-updatecode
drush @sites updatedb

   The pm-updatecode command only updates files, and does not run database update. So with these commands I am first updating modules from site1, then running database update on all sites, then update modules of site2, and run database update on all sites again. Running drush @sites updatedb multiple times, even when there are no updates, should be safe. Take note of any warnings/errors reported, you will want to fix them later, for example:

   WARNING: Updating core will discard any modifications made to Drupal core files, most noteworthy among these are .htaccess and robots.txt. If you have made any modifications to these files, please back them up before updating so that you can re-create your modifications in the updated version of the file.

7. Disable maintenance mode. Cleaning the cache seems unnecessary, as updatedb command does that. drush @sites variable-set site_offline 0.
8. Finalize: re-enable anything disabled before the updates, fix warnings/errors you noted during the update.

This worked well for me, and I hope it works well for you.

• First comes make itself – the oldest and the most widely used software build tool. Stable and powerful. Still, even people who got used to using make, have some gripes about it. The most detailed list of gripes is probably here.
• SCons is a build tool written in Python. I guess I like that “configuration files are Python scripts” – maybe knowing Python is enough to use SCons, which makes SCons a better choice than make for me. SCons seems to have gained some support (scroll down for comments/discussion). There were some doubts about SCons performance (1, 2, and 3); not sure where SCons is at right now in that regard.
• waf, a Python-based framework for configuring, compiling and installing applications.
• pyDoIt is a Python automation tool. It seems to use Python syntax. It aims at bringing the power of build-tools to execute any kind of task, where a task describes some computation to be done (actions), and contains some extra meta-data. Based on the description alone, I’m quite intrigued! I wonder if anyone had already worked with pyDoIt and can share experiences?…
• Rake – Ruby make – is a simple build program with capabilities similar to those of make. Had seen a lot of positive feedback about this one – mostly regarding simplicity of use. Still [py]DoIt so far looks more attractive to me personally.
• Ruffus is a lightweight python module for running computational pipelines. Sounds like some good competition to [py]DoIt!
• Anduril is an open source component-based workflow framework for scientific data analysis. Sounds promising, though the latest downloadable version is over 400 MBs… It probably already contains a bunch of binaries and maybe even data and complete workflows for data analysis. Probably worth a look, but may turn out a little overweight for simple pipelining.
• snakemake is a scalable bioinformatics workflow engine. I get the feeling that Python is truly dominating the pipelines/workflows world: snakemake, as even the name suggests, is in Python, too. The front-page example is so simple and clear, that snakemake immediately pushes DoIt down from the 1st place! Awesome.
• Paver is a yet-another Python-based software project scripting tool along the lines of Make or Rake, designed to help out with repetitive tasks with the convenience of Python’s syntax. Sounds similar to DoIt. Have no idea how they actually compare to each other.

That is it for now.

What were your experiences with automating repetitive tasks and building simple pipelines?

My personal choice was RStudio: it is good-looking, intuitive, easy-to-use, while powerful.

Next step would be using some R-equivalent of the excellent ipython’s Mathematica-like Notebook webinterface…

Redmine is also really cool, and is actually more feature-reach than what BitBucket has to offer, but maintaining it needs just a tiny bit more time and attention than I’m willing to spend these days. So, migration it is!

Redmine has issue 3647 titled “Data import/export system”; it is not resolved, but has a number of links to other resources. Like the redmine exporter at hostedredmine.com, which provides free hosted redmine service. Redmine itself has REST API, though I have no idea if it allows exporting all the data I may need. There’s also an XLS export plugin, but it has to be installed first, and I’m too lazy There’s also TaskAdapter, but they do not support BitBucket (yet?).

For the complete backup, I think of using the pure-ruby redmine project data export script. To migrate issues only, I’ll consider the redmine2bitbucket script.

P.S. Not implying anything (yet?), but my previous migration was from Trac to Redmine… At that time, Trac seemed to have less features than I wanted. And now I’m migrating back to “less features”, but with a benefit of no support required from me.

NetworkX is a pure Python implementation, where anything can be nodes. Both nodes and edges have attributes. NetworkX supports directed graphs and multigraphs (where there are multiple edges between nodes). It might be slower than other implementations, but you may even not notice that – especially when working with smaller graphs, or not applying computationally-intensive algorithms to your graphs.

graph-tool uses the Boost graph library (C++), so it should be really fast. It could be the only multi-threaded graph library for Python. It supports pickling the graphs, allows interactive graph drawing, and has well-illustrated documentation. If performance and efficiency are of utmost importance – could be the best choice.

igraph is also really fast – just like graph-tool when using 1 CPU; graph-tool only wins conclusively when it is run on multiple CPUs/cores. igraph has an R package bindings to C.

Pure python is also an option for really smaller cases.

Finally, there’s a discussion around Python Graph API to simplify the inter-changeability and inter-operability of various existing Python graph modules. It also has a list of some less-known Python graph libraries, so check it out.

There are already quite a few internet resources discussing “python set vs list”, but probably the simplest while elegant way to test that is below.

First, test the speed of adding/appending to a set or a list (here, I’m mimicking the real-life application, thus the test case has an optional loop):

As we can see, set and list are comparable in the speed of adding new items, with list being slightly (~12%) faster than set.

Now, the speed of membership testing: ‘x in tmp’. For this test, I’ve deliberately chosen an imbalance of True (1%) and False (99%) results for the test – again, mimicking the real problem I have at hand:

List is much slower for membership testing, while sets were designed to be fast for doing just that.

• Jython – Python for the Java platform, compile your python scripts into Java bytecode
• Groovy – not Python, but still a scripting language which compiles to jars

• if you don’t have the python-stdeb package: sudo aptitude install python-stdeb
• pypi-install pycassa

That’s it.

Refer to stdeb readme for more information. You will need that if there are dependencies – which might not be resolved automatically by stdeb.

Before stdeb, it wasn’t exactly trivial to make a .deb from python module.

• Type your code in the lower pane of the window. To execute, press Ctrl+Enter. One-liners can be executed by simply pressing Enter; If you don’t want them executed, press Space and then Enter.

• Use Ctrl+Up and Ctrl+Down to navigate between code segments you’ve already executed. You can write a few letters before pressing Ctrl+Up, and DreamPie will only search through code segments starting with those letters.

• Press Tab or Ctrl+Space to show a list of completions to the current expression. It will also complete file names!

• Your results are stored in variables named _0, _1, and so on.

• Type a function name and press the space key and DreamPie will automatically add parentheses for you!

Here’s a simple R script to do just that. It is abundantly commented, and also contains an optional (commented out) fragment which allows the removal of more low-variance, low-intensity probesets.

Hint: click the “plain text” box header to be able to right-click the code, “Select All”, and then “Copy”.
[CODE]
filter_below_spikes = function(eset) {
# Finds max(lowest) AFFX/spike-in intensity, and removes rows consisting entirely of values below max(lowest).
# @param eset
# ExpressionSet
# @returns
# exprs(ExpressionSet), filtered

# Without Biobase exprs() will not work.
require(Biobase)
expr = exprs(eset)

# ‘expr’ sample:
# 1
# 1367452_at 10.880208
# 1367453_at 10.554647

cat(nrow(expr), “rows before filtering.\n”)

# Make a vector of spike row names.
spikes = grep(“AFFX”, rownames(expr), value = TRUE)
cat(“Expression matrix has”, length(spikes), “spike-in rows.\n”)
cat(“Summary of spike-in values distribution follows:\n”)
print(summary(expr[spikes, ]))

# Find max(lowest) spike-in values.
minval_max = max(as.double(substr(grep(“Min”, summary(expr[spikes, ]), value = TRUE), 10, 14)))
cat(“max(minimal spike-in log-intensity values) =”, minval_max, “\n”)

# Remove spike-ins from expr.
expr = expr[grep("AFFX", rownames(expr), value = TRUE, invert = TRUE), ]

nospike_rows = nrow(expr)
cat(nospike_rows, “rows remaining after the removal of”, length(spikes), “spike-in probesets.\n”)

# Optional: calculate max(SD) of all removed rows.
#bad_sds_max = max(apply(expr[!apply((expr > minval_max), 1, any),], 1, sd))

# Now remove all rows, where each value is <= minval_max. expr = expr[!apply((expr <= minval_max), 1, all), ] cat(nrow(expr), "rows remaining after filtering out", nospike_rows - nrow(expr), "probesets with all values below", minval_max, "\n") #cat(bad_sds_max, "is max(SD) of all", nospike_rows - nrow(expr), "filtered probesets with all values below", minval_max, "\n") # Optional: Remove *some* of the rows, which have at least one value below minval_max, and row_SD <= bad_sds_max. #pre_final_rows = nrow(expr) #expr = expr[(apply(expr, 1, sd) > bad_sds_max) | (apply(expr, 1, min) > minval_max), ]
#cat(pre_final_rows-nrow(expr), “rows with SD <=", bad_sds_max, "and min(row) <=", minval_max, "were removed.\n") #cat(nrow(expr), "final rows returned.\n") return(expr) } [/CODE] Sample use:

> source(“script.R”)
> expr.filtered = filter_below_spikes(eset)

Comments and suggestions are welcome.

Slow DoS attack with just 1 computer against a number of web servers, including Apache: slowloris. There is a solution for Apache, packaged for RedHat and also available for Debian.

Finally, there’s Go programming language. The most inspiring promise to me personally is the ease of execution parallelization with language’s built-in syntactic constructs. That is something highly desired. Also, I like that it is a compiled language. However, it might be 10%-20% slower than pure C. Let’s see how it grows.

Development: see http://www.ruzee.com/blog/2009/01/apache-virtual-hosts-a-clean-setup-for-php-developers. This setup works very well, unless you need to create several virtual hosts every day – in which case necessary actions could be partially scripted.

Miscellaneous tools exist to ease this task. This post was inspired by redet’s comparison of regexp helper tools – it could be sufficient to read only that, if you’re going to try the mentioned tools yourself. Otherwise, read on.

Kodos is an easy-to-use (but powerful enough) Python tool to verify regular expressions against sample user-provided strings. Available for both Linux and Windows. Generates ready-to-use Python regexp code.

redet assumes some knowledge of REs. Unlike all the other helper tools, it has the widest code generation coverage – just go to “Tools->Save command information” to generate code (Python, Perl, C, grep – just to name a few of the 57 (!) languages/tools/editors supported). At the same time, redet does not depend on specific versions of all supported RE engines – instead, redet automatically tests for features in available versions.

visual-regexp is similar to kodos in features.

txt2regex is a regex wizard written in bash, capable of generating code for 24 languages/editors/programs (by default, these are enabled: perl, php, postgres, python, sed and vim). It is really simple and convenient, though some understanding of “groups” and “character ranges” is highly recommended.

KRegExpEditor was a part of kde3-utils, and seems to be missing from current KDE 4.3. It is similar to kodos and visual-regexp in its philosophy and features.

I would recommend using one of kodos/visual-regexp/kregexpeditor to quickly test or design your regular expression, then use redet to generate appropriate source code. As an alternative (and when working in a console), just use txt2regexp to generate the code you need.

Tools I haven’t tried include regex coach and regexpviewer.

Please also see this extensive list of regular expression and pattern-matching related software.

However, version 0.16.1 is very unstable, and dies with

***MEMORY-ERROR***: screem[5527]: GSlice: assertion failed: sinfo->n_allocated > 0

As a workaround (initially suggested for the highly similar Firestarter crashes), try running screem with this command:

G_SLICE=always-malloc screem

Too bad last development version of Screem is dated March 2006.

If you happen to be developing a C/C++ program – do use mtrace and valgrind, those are huge helpers against the problems akin to that shown on the graphs.

I’m sure that you are aware of the fact that floats representation in any programming language is limited by the precision of the internal binary representations. In other words, you can never have an exact float representation – there will always be some precision associated with the float you are working with. The simplest example is the difference in precision between the float and double types in C.

Suppose I have the following code fragment:
[C] if ( result.score >= input->raw_cut_off ) [/C]

Both result.score and input->raw_cut_off are of type float, and can have positive and negative values. When compared with the greater than or equal ( >= ) operator, it is not always that condition is true – for the precision reasons shortly mentioned above.

As I already said, there is no precision specification for equality operators in C. But it is quite simple to “invent” precision specification; e.g. if I wanted to test for equality only, I could write
[C] if ( fabsf( result.score – input->raw_cut_off ) < 0.000001 )[/C] In this example, I'm effectively asking for 6-digit precision for the equality comparison of floating-point values. Note, that if you replace that 0.000001 with the actual precision limit of the floating type you are using, you will be "exactly" comparing floating-point numbers - up to that type's precision, of course .

The first-most example with the >= operator can be rewritten as
[C] if ( result.score > ( input->raw_cut_off – precision) ) [/C]
where precision is exactly what it is named, e.g. precision = 0.000001.

Sources used:

• comment by Randy A. Ynchausti
• scientific programming