Autarchy of the Private Cave » database https://bogdan.org.ua Tiny bits of bioinformatics, [web-]programming etc Wed, 28 Dec 2022 16:09:04 +0000 en-US hourly 1 https://wordpress.org/?v=3.8.27 Weird orthology species names in Ensemblhttps://bogdan.org.ua/2008/09/30/weird-orthology-species-names-in-ensembl.html https://bogdan.org.ua/2008/09/30/weird-orthology-species-names-in-ensembl.html#comments Tue, 30 Sep 2008 11:01:59 +0000 http://bogdan.org.ua/?p=398 For the COTRASIF tool, I’ve been using the Ensembl Compara database (since release 47) to automatically import into COTRASIF gene orthology mappings.

However, with the E!50 release, the Compara database was dropped.

Looking for another option to get orthologs from Ensembl (using martservice, via biomart.org), I tried using the standard query – selecting “Homologs” group on the “Attributes” page for a single species database, and then selecting appropriate second species to get orthology mappings.

Imagine my surprise, when not only in the interface, but also in the generated XML file I found attribute names like “cow_ensembl_gene” :-O

I only need 11 species at the moment, and excluding the sufficiently unique name mappings like “zebrafish – danio rerio”, there is a number of questionable mappings: “yeast” for S. cerevisiae (could be S.pombe), “rat” for R. norvegicus (could be R.rattus), “anopheles” for A.gambiae (could be some other Anopheles). Other mappings might be also non-unique, especially for people working with different species of the same genus.

Am I missing some system in this naming “convention”, or am I the only one who finds it strange?

Is there a way not to use “common species names” when importing orthology data from Ensembl with the help of martservice?

Share

]]> https://bogdan.org.ua/2008/09/30/weird-orthology-species-names-in-ensembl.html/feed 0 Convert MySQL database from one encoding/collation into anotherhttps://bogdan.org.ua/2008/02/08/convert-mysql-database-from-one-encodingcollation-into-another.html https://bogdan.org.ua/2008/02/08/convert-mysql-database-from-one-encodingcollation-into-another.html#comments Fri, 08 Feb 2008 19:17:45 +0000 http://bogdan.org.ua/2008/02/08/convert-mysql-database-from-one-encodingcollation-into-another.html Most frequent use: convert database from latin1_swedish to utf8_general_ci.
Original script found at: MySQL and UTF-8.

Update: the original script had an error, it would generate queries likes this one (note the bold part):

ALTER TABLE `links` CHANGE `link_rel` `link_rel` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL DEFAULT NULL;

This is clearly wrong syntax (and logic). I fixed this by making comparison to NULL strict (with three equal signs instead of two):

// Does the field default to null, a string, or nothing?
if ($row['Default'] === NULL)

Update 2: based on comment by banesto, I modified the script; now it does not require specifying the from_collation, it’s sufficient to specify to_collation (which will be used for all the fields and tables). The modified code is:

if ($row['Collation'] == ” || $row['Collation'] == $convert_to)
continue;

Update 3: the long-lasting, re-appearing NOT NULL DEFAULT NULL problem is finally fixed.

Update 4: incorporated Russ’s fix to skip numeric fields (in order to leave autoincrement values intact).

Here’s the script itself: (to copy-paste: first click the “Plain text” header)

  1. <?php
  2. // Original script (v1.0) by/from: http://www.phpwact.org/php/i18n/utf-8/mysql
  3. // Improved/modified (v1.05) by Bogdan http://bogdan.org.ua/
  4. // Last updated: 2010-06-28
  5.  
  6. // this script will output all queries needed to change all fields/tables to a different collation
  7. // it is HIGHLY suggested you make a MySQL backup prior to running any of the generated queries
  8.  
  9. // this code is provided AS IS and without any warranty
  10.  
  11. // add text/plain header when used not as a CLI script; PHP CLI SAPI shouldn't output headers
  12. header("Content-Type: text/plain");
  13.  
  14. // precaution
  15. die("Make a backup of your MySQL database, then remove this line from the code!");
  16.  
  17. set_time_limit(0);
  18.  
  19. // collation you want to change to:
  20. $convert_to   = 'utf8_general_ci';
  21.  
  22. // character set of new collation:
  23. $character_set= 'utf8';
  24.  
  25. // DB login information - *modify before use*
  26. $username = 'user';
  27. $password = 'pass';
  28. $database = 'database_name';
  29. $host     = 'localhost';
  30.  
  31. //-- usually, there is nothing to modify below this line --//
  32.  
  33. // show TABLE alteration queries?
  34. $show_alter_table = true;
  35. // show FIELD alteration queries?
  36. $show_alter_field = true;
  37.  
  38. mysql_connect($host, $username, $password);
  39. mysql_select_db($database);
  40.  
  41. $rs_tables = mysql_query(" SHOW TABLES ") or die(mysql_error());
  42.  
  43. while ($row_tables = mysql_fetch_row($rs_tables)) {
  44.     $table = mysql_real_escape_string($row_tables&#91;0&#93;);
  45.    
  46.     // Alter table collation
  47.     // ALTER TABLE `account` DEFAULT CHARACTER SET utf8
  48.     if ($show_alter_table)
  49.         echo("ALTER TABLE `$table` DEFAULT CHARACTER SET $character_set;\n");
  50.  
  51.     $rs = mysql_query(" SHOW FULL FIELDS FROM `$table` ") or die(mysql_error());
  52.  
  53.     while ( $row = mysql_fetch_assoc($rs) ) {
  54.        
  55.         if ( $row&#91;'Collation'&#93; == '' || $row&#91;'Collation'&#93; == $convert_to )
  56.            continue;
  57.  
  58.         // Is the field allowed to be null?
  59.         if ( $row&#91;'Null'&#93; == 'YES' )
  60.            $nullable = ' NULL ';
  61.         else
  62.             $nullable = ' NOT NULL ';
  63.  
  64.         // Does the field default to null, a string, or nothing?
  65.         if ( $row&#91;'Default'&#93; === NULL && $row&#91;'Null'&#93; == 'YES' )
  66.            $default = " DEFAULT NULL ";
  67.         elseif ( $row&#91;'Default'&#93; != '' )
  68.            $default = " DEFAULT '".mysql_real_escape_string($row&#91;'Default'&#93;)."'";
  69.        else
  70.             $default = '';
  71.  
  72.         // sanity check and fix:
  73.         if ($nullable == ' NOT NULL ' && $default == ' DEFAULT NULL ') {
  74.             $default = '';
  75.             echo "/* Warning: wrong combination of 'default value' and 'NULL-flag' detected - and fixed! */\n";
  76.             echo "/* Diagnostics: row&#91;Null&#93; = '$row&#91;Null&#93;', row&#91;Default&#93; = " . mysql_real_escape_string($row&#91;'Default'&#93;) . ", MySQL version: " . mysql_get_server_info() . " */\n";
  77.        }
  78.  
  79.         // Don't alter INT columns: no collations, and altering them drops autoincrement values
  80.         if (strpos($row&#91;'Type'&#93;, 'int') !== false) {
  81.            $show_alter_field = False;
  82.         }
  83.         else {
  84.             $show_alter_field = True;
  85.         }
  86.  
  87.         // Alter field collation:
  88.         // ALTER TABLE `tab` CHANGE `field` `field` CHAR( 5 ) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL
  89.         if ($show_alter_field) {
  90.             $field = mysql_real_escape_string($row&#91;'Field'&#93;);
  91.            echo "ALTER TABLE `$table` CHANGE `$field` `$field` $row&#91;Type&#93; CHARACTER SET $character_set COLLATE $convert_to $nullable $default;\n";
  92.         }
  93.     }
  94. }
  95. ?>

Share

]]> https://bogdan.org.ua/2008/02/08/convert-mysql-database-from-one-encodingcollation-into-another.html/feed 60 MySQL: INSERT IF NOT EXISTS syntaxhttps://bogdan.org.ua/2007/10/18/mysql-insert-if-not-exists-syntax.html https://bogdan.org.ua/2007/10/18/mysql-insert-if-not-exists-syntax.html#comments Thu, 18 Oct 2007 13:20:21 +0000 http://bogdan.org.ua/2007/10/18/mysql-insert-if-not-exists-syntax.html To start: as of the latest MySQL, syntax presented in the title is not possible. But there are several very easy ways to accomplish what is expected using existing functionality.

There are 3 possible solutions: using INSERT IGNORE, REPLACE, or INSERT … ON DUPLICATE KEY UPDATE.

Imagine we have a table:

  1. CREATE TABLE `transcripts` (
  2.  `ensembl_transcript_id` varchar(20) NOT NULL,
  3.  `transcript_chrom_start` int(10) unsigned NOT NULL,
  4.  `transcript_chrom_end` int(10) unsigned NOT NULL,
  5.  PRIMARY KEY  (`ensembl_transcript_id`)
  6. ) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Now imagine that we have an automatic pipeline importing transcripts meta-data from Ensembl, and that due to various reasons the pipeline might be broken at any step of execution. Thus, we need to ensure two things: 1) repeated executions of the pipeline will not destroy our database, and 2) repeated executions will not die due to ‘duplicate primary key’ errors.

Method 1: using REPLACE

It’s very simple:

  1. REPLACE INTO `transcripts`
  2. SET `ensembl_transcript_id` = 'ENSORGT00000000001',
  3. `transcript_chrom_start` = 12345,
  4. `transcript_chrom_end` = 12678;

If the record exists, it will be overwritten; if it does not yet exist, it will be created.
However, using this method isn’t efficient for our case: we do not need to overwrite existing records, it’s fine just to skip them.

Method 2: using INSERT IGNORE
Also very simple:

  1. INSERT IGNORE INTO `transcripts`
  2. SET `ensembl_transcript_id` = 'ENSORGT00000000001',
  3. `transcript_chrom_start` = 12345,
  4. `transcript_chrom_end` = 12678;

Here, if the ‘ensembl_transcript_id’ is already present in the database, it will be silently skipped (ignored). (To be more precise, here’s a quote from MySQL reference manual: “If you use the IGNORE keyword, errors that occur while executing the INSERT statement are treated as warnings instead. For example, without IGNORE, a row that duplicates an existing UNIQUE index or PRIMARY KEY value in the table causes a duplicate-key error and the statement is aborted.”.) If the record doesn’t yet exist, it will be created.

This second method has several potential weaknesses, including non-abortion of the query in case any other problem occurs (see the manual). Thus it should be used if previously tested without the IGNORE keyword.

There is one more option: to use INSERT … ON DUPLICATE KEY UPDATE syntax, and in the UPDATE part just do nothing do some meaningless (empty) operation, like calculating 0+0 (Geoffray suggests doing the id=id assignment for the MySQL optimization engine to ignore this operation). Advantage of this method is that it only ignores duplicate key events, and still aborts on other errors.

As a final notice: this post was inspired by Xaprb. I’d also advise to consult his other post on writing flexible SQL queries.

Share

]]> https://bogdan.org.ua/2007/10/18/mysql-insert-if-not-exists-syntax.html/feed 46