change-collation-on-all-tables-and-columns-in-mysql

Thanks , good work

SQL query:

ALTER TABLE `node_revisions` CHANGE `body` `body` longtext CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL ;

MySQL said:

#1283 – Column ‘body’ cannot be part of FULLTEXT index

thank you for contributing, I’ve updated the script.

There’s still an error with some fields that say: “NOT NULL DEFAULT NULL”

Here’s a fix:

// Does the field default to null, a string, or nothing?
if ( $row['Default'] === NULL )
$default = $row['Null'] == ‘YES’ ? ” DEFAULT NULL ” : “”;
elseif ( $row['Default'] != ” )
$default = ” DEFAULT ‘”.mysql_real_escape_string($row['Default']).”‘”;
else
$default = ”;

I have a simple way:

First export DB to file .sql then find “latin1″ and replace all as “utf8″ with your text editor (i’m use text wrangler).
Second, run this script:

ALTER DATABASE `your DB name` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci

Import file .sql for the final step.

That ’s all, very simple. By this way all old data will by change to UTF-8. Have a nice day.

I’ve added text/plain header, that makes this script cleaner indeed.

I have one tiny thing I changed which was a CLI switch. It does not print “< pre >” unless run over HTTP (Line 38):

if ($_SERVER['HTTP_HOST']){
print ‘< pre >‘;
}

An alternative to this is to just output at “Content-type: text/plain”. That will tell the browser to print newlines.

Seeya…


// Does the field default to null, a string, or nothing?
if ( $row['Null'] == 'YES' && $row['Default'] === NULL )
$default = " DEFAULT NULL";
elseif ( $row['Default'] != '' )
$default = " DEFAULT '".mysql_real_escape_string($row['Default'])."'";
else
$default = '';


will fix null problem

Looking at the explanation of error #1283:

Error: 1283 SQLSTATE: HY000 (ER_BAD_FT_COLUMN)
Message: Column ‘%s’ cannot be part of FULLTEXT index

it looks like the problem might be with those specific columns, not with the change of collation.

I guess the source of these problem is that ALL columns in fulltext index should have same collation and error appears when we change collation of one of the columns contained in full text index.
The only solution that I know is to drop index , change collation and recreate index again. But it would be nice if somebody can write the script autоmating this …

Wouldn’t any new tables being created would still get created with the old collation if the database wasn’t altered wouldnt it?

I also once read somewhere that you have to create a completely new database for it to work properly but perhaps that was only versions of MySQL below 5.x

I hope I understood your question correctly.

ALTER DATABASE `dbname` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci

Should you first create a new DB and copy the tables to it in this new format?

Error: 1283 SQLSTATE: HY000 (ER_BAD_FT_COLUMN)
Message: Column ‘%s’ cannot be part of FULLTEXT index

it looks like the problem might be with those specific columns, not with the change of collation.

1. did you really mean to say that your MySQL client is at version 4.1.22, and Server’s Version is 5.0.67? this is weird, and if it is really the case, then it could be causing you miscellaneous troubles.

2. instead of converting the database, try to set it to utf8_general_ci *before* Joomla installation; if Joomla allows that – ask it to install with utf8 support in DB from the very beginning. If Joomla doesn’t support that – I would suggest that you try Drupal 6.x, it might work perfectly out of the box for you.

3. the errors you list look like you are trying to feed the PHP script to MySQL…. are you accessing the script with your browser, or just feeding it verbatim to the mysql?

I’ve tried the scriptd by Bogdan and Russ as well as the link supplied by Bogdan at the top. Since I really know nothing about MySQL I do not understand how to fix the error messages I’m getting:

1. FIRST ERROR at top and down.)

ERROR: Unknown Punctuation String @ 1
STR: <?
SQL: <?php

// this script will output the queries need to change all fields/tables to a different collation
// it is HIGHLY suggested you take a MySQL dump prior to running any of the generated
// this code is provided as is and without any warranty

set_time_limit(0);<?php

2. SECOND ERROR at bottom)

SQL query:

<?php // this script will output the queries need to change all fields/tables to a different collation // it is HIGHLY suggested you take a MySQL dump prior to running any of the generated // this code is provided as is and without any warranty set_time_limit(0);

MySQL said: Documentation
#1064 – You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘<?php

// this script will output the queries need to change all fields/table’ at line 1

<?php
// original script (v1.0) by/from: http://www.phpwact.org/php/i18n/utf-8/mysql
// improved/modified (v1.02) by Bogdan Tokovenko http://bogdan.org.ua/
// this script will output all queries needed to change all fields/tables to a different collation
// it is HIGHLY suggested you take a MySQL dump prior to running any of the generated queries
// this code is provided AS IS and without any warranty
set_time_limit(0);

// collation you want to change to:
$convert_to = ‘utf8_general_ci’;
// character set of new collation:
$character_set= ‘utf8′;
$show_alter_table = true;
$show_alter_field = true;
// DB login information – modify before use
$username = ‘username’;
$password = ‘password’;
$database = ‘database_name’;
$host = ‘localhost’;
//– nothing to modify below this line –//
mysql_connect($host, $username, $password);
mysql_select_db($database);
$rs_tables = mysql_query(” SHOW TABLES “) or die(mysql_error());
print ”;
while ($row_tables = mysql_fetch_row($rs_tables)) {
$table = mysql_real_escape_string($row_tables[0]);
// Alter table collation
// ALTER TABLE `account` DEFAULT CHARACTER SET utf8
if ($show_alter_table)
echo(“ALTER TABLE `$table` DEFAULT CHARACTER SET $character_set;\r\n”);
$rs = mysql_query(” SHOW FULL FIELDS FROM `$table` “) or die(mysql_error());
while ($row=mysql_fetch_assoc($rs)) {
if ($row['Collation'] == ” || $row['Collation'] == $convert_to)
continue;
// Is the field allowed to be null?
if ( $row['Null'] == ‘YES’ )
$nullable = ‘ NULL ‘;
else
$nullable = ‘ NOT NULL’;
// Does the field default to null, a string, or nothing?
if ( $row['Default'] === NULL )
$default = ” DEFAULT NULL”;
else if ($row['Default']!=”)
$default = ” DEFAULT ‘”.mysql_real_escape_string($row['Default']).”‘”;
else
$default = ”;
//don’t alter int columns as they do not have a collation anyway and it buggers up the extra parameters
if (strpos($row[Type], ‘int’) !== false)
$show_alter_field = false;
else
$show_alter_field = true;
// Alter field collation:
// ALTER TABLE `tab` CHANGE `fiel` `fiel` CHAR( 5 ) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL
if ($show_alter_field) {
$field = mysql_real_escape_string($row['Field']);
echo “ALTER TABLE `$table` modify `$field` $row[Type] COLLATE $convert_to ; \r\n”;
}
}
}
?>

Chad

(The problem is that I can’t reproduce this error on my databases – thus “steps to reproduce” must include a sample of actual database table.)

it is the same old problem which is for some reason happening all the time.

I’ve just tested the script locally – and for me it works perfectly.

I can only recommend that after you get all the generated queries, just run a global search-and-replace, replacing each and every “NOT NULL DEFAULT NULL” with “NOT NULL”

If you would like the script to do that for you (and also contribute to let others avoid this error), please do the following:
- for the first table where you get the error, run the query

SHOW FULL FIELDS FROM tablename

(where tablename is the name of that troublesome table),
and post the result here. That might help me identify the reason why it is not working for you but does work for me.

i get on every table

ALTER TABLE `banners` CHANGE `description` `description` varchar( 255 ) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL DEFAULT NULL ;

MySQL said: Documentation
#1067 – Invalid default value for ‘description’ ”

the original installed field values here are

field: description
type: text
collation: utf8_unicode_ci
attributes: (empty)
null : No
default: (empty)
extra: (empty)

This is basically fact on all tables and it runs into this error on same fields and stops. Any ideas would be welcome?

Chad

I’ve corrected the text – of course, it must be database name.

in the line below, do I enter each table individually or simply put the database name in the place of ‘table’?

$database = ‘table’;

thanks

There are probably many people that can explain the details of why you do it this way. But what I know is that if you can to “convert” existing data from one Character set to another in MySQL, you do it by making the field a binary (and thus tricking MySQL to reinterpret the raw data stored when you bring it back into a text field in another character set).

I have implemented two such conversions for two common situations. But you can play around with this quite a bit if you have some really messed up data.

why would you need these conversions?

char, varchar and text -> blob
mediumtext -> mediumblob
longtext -> longblob

blobs are for binary data, so why put the text into binary fields?