wiki.blueonyx.it

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== BlueOnyx General Migration Utility (GMU) ======

===== Brief Summary =====

The BlueOnyx General Migration Utility (GMU) has been created to allow easy migrations from non-BlueOnyx servers to BlueOnyx 5209R. For this purpose it uses a custom Perl script to gather as much needed info as it can from a non-BlueOnyx server and then builds an XML file and tarballs that contain all info that is needed to perform an import on BlueOnyx 5209R.

On BlueOnyx 5209R we do have a script that can perform the import. This guide explains the usage.

Please note: Currently the only supported platform for exports to BlueOnyx are Verio FreeBSD and Verio CentOS-4 servers that run the Verio GUI. Support for further platforms will be added later (think Plesk & cPanel).

Additionally: MySQL will **not** be exported or imported. You will need to move that manually.

===== How to export: =====

==What you need:==

  * '//root//' access to a supported export platform (Verio FreeBSD 6.4 or Verio CentOS 4.9)
  * A couple of GB of free space on that server.

==Setup of the exporter:==

Login as 'root' on the server on which you want to generate the export. Or login as user with "wheel" access and perform "//su -//" to gain 'root' access.

Make sure you have enough disk space for the export and note which partition has the most space. This usually ought to be /home/. You can check the available disk space with the command "//df -h//".

Download the migration utility:

  cd /home
  wget http://devel.blueonyx.it/pub/BlueOnyx/TAR/Verio-to-BlueOnyx-5209R-exporter.tar.gz
  tar zxvf Verio-to-BlueOnyx-5209R-exporter.tar.gz
  cd Verio-to-BlueOnyx-5209R-exporter/migrator

Inside that directory is the script '//verio-export-to-blueonyx.pl//', which is the only script you will need to run during the export.

To run the script you prefix it with './' - Example:

  export_box# ./verio-to-blueonyx-exporter.pl -h
  ##################################################################### 
  # verio-to-blueonyx-exporter.pl: BlueOnyx Generic Migration Utility #
  #####################################################################
  # 
  # Verio (FreeBSD and CentOS) to BlueOnyx 5209R exporter.
  # This utility parses the Apache and Sendmail config (and 
  # cpx.conf) to generate an XML file and Tarballs that allow
  # you to import all Vsites and Users on BlueOnyx 5209R.
  
  usage:   verio-export-to-blueonyx.pl [OPTION]
           -c export configuration only
           -d export dir (assumes /home/Verio-to-BlueOnyx-5209R-exporter/migrator/data/) unless specified
           -s SCP target details (like 'root@host:/dir/')
           -h help, this help text
  
  export_box#

The command line parameter '-h' displays the above help text. 

===Explanation of the command line options:===

^ Switch      ^ Value       ^ Meaning          ^
| -c  | <none>     | If you do not want to export any TarBalls with Vsite and User data, then you can use the '-c' switch. This will only export the configuration and information about Vsites and Users.        |
| -d  | <local directory> | This allows you to specify where the exported XML (and TarBalls) will be stored on the export server. If not specified, then the 'data' directory inside the directory where the script resides will be used.|
| -s  | <root@target.com:/dir/> | If the server on which you generate the export does not have enough diskspace to store all TarBalls, then you can use the '-s' switch and can specify the full username, hostname and directory on a remote server. During export every TarBall is then generated locally and right after creation this TarBall is sent via 'scp' command over SSH to the remote server where you want to import. After the transmission the locally stored TarBall is deleted to conserve space. **Please note:** For this to work you **need** to exchange SSH keys. The export server must be allowed to login (w/o password) via SSH key on the target server. |
| -h  | <none>     | Display the help text and exit.   |


The exporter itself is very verbose and will tell you at ever step and turn what it does.

If you want to save the output of the exporter to a textfile for later review (without missing anything that goes on during the export), then you would perform the export this way:

  export_box# ./verio-to-blueonyx-exporter.pl 2>&1 | tee export.log

In that case a log of all status and error messages will be written to 'export.log'.

==Perform the Export:==

To do the actual export run one of the two commands:

**To store export locally:**

  export_box# ./verio-to-blueonyx-exporter.pl 2>&1 | tee export.log

**To store export remotely:**

On the remote server add '/root/.ssh/id_rsa.pub' from the export server to '/root/.ssh/authorized_keys'. Then create the import directory there: 

  import_box# mkdir /home/import

Go back to the server where you want to perform the export. Start the exporter there:

  export_box# ./verio-to-blueonyx-exporter.pl -s root@10.1.254.1:/home/import/ 2>&1 | tee export.log

This will take a while. The progress will be shown on the screen.


===== How to import: =====

==What you need:==

  * '//root//' access to a supported import platform (BlueOnyx 5209R)
  * Enough disk space to store **and** import all data.

==Setup of the importer:==

The RPM that contains the importer is named 'blueonyx-gmu' and is available in the BlueOnyx 5209R YUM repository. To install it on a 5209R run the following command:

  yum install blueonyx-gmu

This will install 'blueonyx-gmu' and all dependencies. After the install of GMU there is only one script that you need to run the install:

  import_box# /usr/sausalito/sbin/blueonyx-import.pl -h
  ########################################################## 
  # blueonyx-import.pl: BlueOnyx Generic Migration Utility #
  ##########################################################
  
  usage:   blueonyx-import.pl [OPTION]
           -c import configuration only
           -d directory of that contains the exported files
           -i import all virtual sites with this IP address
           -n import only these sites ie -n "ftp.foo.com,www.bar.com"
           -r remove items ie "-r (vsites|users|resellers)"
           -a skip import of all Reseller accounts
           -e just examine and verify dump
           -q Fix Vsite and User quota to OK limits
           -h help, this help text
  
  import_box# 
  
The command line parameter '-h' displays the above help text.

===Explanation of the command line options:===

^ Switch             ^ Value       ^ Meaning          ^
| -c  | <none>     | If you do **not** want to import any Vsites and User Tarballs, then use this switch. That will only create the Reseller, Vsite and User accounts and will import their configuration. |
| -d  | <directory>     | This allows you to specify the directory where the TarBalls and the XML from the export server are located on the import server. |
| -i  | <IP Address>     | If you need to change the IP Address of Vsites during the import, then you can specifiy the new IP Address of Vsites on the command line via this parameter. If not specified, then Vsites will be imported onto the same IP Address that they had during the export. |
| -n  | <"domain1.com,domain2.net">     | If you want to import only one (or a few) Vsites (and all their Users), then you can use this option. |
| -r  | <vsites><users><resellers>     | If an import failed for any reason, then you can use '-r vsites' to remove all Vsites (and Users). If you want to only remove all Users (not System-Administrators!), then you can use '-r users'. If you use '-r resellers', then all Reseller accounts (including their Vsites and Users!) will be removed. |
| -a  | <none>     | If specified, no Reseller accounts will be created during import. |
| -s  | <none>     | Vsite that already exists on server doesn't get imported. |
| -e  | <none>     | Does not perform an import. Instead it will check the integrity of the XML file and the TarBalls in the specified import directory. |
| -q  | <none>     | Does not perform an import. If run with this option, then any Vsite or User that is over quota (or at +90% disk usage) will receive enough additional disk space to end the over-quota situation. This option is used automatically at the end of an import. |
| -p  | <none>     | Does not perform an import. This will read the server wide PHP settings and will apply them to all Vsites. |
| -h  | <none>     | Display the help text and exit.   |

**Perform the Import:**

Assume you do have the TarBalls and the BlueOnyx-Migrate.xml in '/home/import/' and you want to perform an import of **all** those Vsites and Users. And you want to change the IP Addresses of all Vsites during the import to '10.1.254.50' In this case you would run the following command on your BlueOnyx 5209R:

  import_box# /usr/sausalito/sbin/blueonyx-import.pl -d /home/import/ -i 10.1.254.50 2>&1 | tee /tmp/import.log

This will perform the following steps:

**Preparational steps:**

  * The importer will check if XML file and TarBalls are present.
  * It will report where and when the export was created.
  * It shows how many Vsites and Users the export contains.
  * It will verify the MD5 checksum of the Tarballs.

**Reseller related steps:**

  * Creation of Reseller accounts (if any are present in the export)

**Vsite related steps:**

  * Creation of Vsites
  * Import of the Vsite TarBalls
  * If SSL was active and certificate and key (and intermediate) were present, then they will be moved to the 'certs' directory of the Vsite. While this should immediately make SSL functional for Vsites with enabled SSL: I couldn't test this. 

**User related steps:**

  * Creation of the Vsite Users
  * Import of the User TarBalls
  * The 'mbox' file will be imported and fixed.
  * If the user used 'Maildir', then this will be converted to 'mbox',

At the end of the import the following correctional steps will be performed automatically:

**Cleanup related steps:**

  * The 'Web-Owner' of Vsites will be set to the dedicated 'siteAdmin' account.
  * The /web directory of Vsites will be chowned to the UID of the 'siteAdmin' and the GID of the Vsite.
  * Allowed disk quota for Vsites and Users will be raised to safe levels if a user is using 90% (or more) of his allowed disk quota. The formula for this is: Set a new quota that is 25% bigger than whatever space the Vsite or User is currently using.

**Diagnostics:**

  * All status and error messages are logged to the logfile that you specified via the appended 'tee' command and are shown on the screen.

===== Troubleshooting =====

Both the export and import scripts are very complex and contain roughly one month of man-hours in coding (packed into an intense two weeks of programming). I did test the exporter and importer extensively. But something might always go wrong and things might not work as intended.

==== During Export ====

Although the exporter is the most complex piece of code: It just should work. If it aborts with an error message for any reason, then get in touch with me and report the error message. Most likely I will need access to the box to diagnose this.

==== During Import ====

The import utility should give you a good indication about the nature of any problem that it did run into. Also see '/var/log/messages' for any CCEd related error messages that happened during CREATE or SET transactions.

**The most probable causes of failure during import are:**

  * /etc/mail/aliases did contain entries that conflict with whatever was being added during User or Vsite creation. Solution: Clean up /etc/mail/aliases to remove dormant or conflicting entries.
  * Username or Vsite name already taken. Solution: Only do an import on a 5209R that doesn't yet contain any Vsites and Users. Use the '-r' option of the importer to remove Vsites and Users before you do an import run.
  * Email- or Web-Alias already taken. The importer takes great care to not set any aliases that are already taken by other Vsites and/or Users. If it detects such a conflict, it will import the Vsite (or User) without the conflicting alias.

**Uncritical messages:**

During imports of TarBalls you might see something like this:

  tar: Ignoriere unbekanntes Schlüsselwort „SCHILY.dev“ für erweiterten Kopfteil
  tar: Ignoriere unbekanntes Schlüsselwort „SCHILY.ino“ für erweiterten Kopfteil
  tar: Ignoriere unbekanntes Schlüsselwort „SCHILY.nlink“ für erweiterten Kopfteil

That can be ignored. The TarBalls will still import fine and this error message (actually it's just a 'notice' and not an error) is caused by drastically different TAR versions and/or MacOS related. Like if a user FTP'ed in stuff with his MacBook.

=== An Import failed. Now what? ===

Check the error message you got. Check /var/log/messages for the SET and CREATE transaction that failed. Try to run that SET or CREATE transaction manually from '/usr/sausalito/bin/cceclient' as that might give you a more detailed error message or further indications about the nature of the problem.

It **could** also be that it was just CCEd that was a bit temperamental during the import and decided to take a dive. 

Once you solved the problem and want to try another import you need to remove all Vsites, Users and Resellers first. To do so, run these two commands:

  /usr/sausalito/sbin/blueonyx-import.pl -r vsites
  /usr/sausalito/sbin/blueonyx-import.pl -r resellers

That will delete all Vsites and all Users and also all Resellers. It will *not* remove the 'admin' account or any GUI created administrative users with the System-Administrator flag set.

Afterwards there are two places you also want to check before attempting another import. Usually the GUI takes care of removing these entries, but sometimes some junk remains around. So please run ...

  cat /etc/mail/aliases

This should be pretty bare. The last good lines of it should read like this:

  # Person who should get root's mail
  #root:          marc
  root:           admin
  root-admin:     admin

If there is anything **below** that (which doesn't relate to a System-Administrator), then delete these lines.

  cat /etc/group

The last couple of lines should read like this:

  sshd:x:74:
  httpd:x:48:
  site-adm:x:1000:
  admin-users:x:1001:

In that case: All good. If it looks like this (**after** you deleted all Vsites), then you have a problem:

  sshd:x:74:
  httpd:x:48:
  site-adm:x:1000:
  admin-users:x:1001:
  site1:x:1027:admin
  site2:x:1028:admin
  site6:x:1032:admin
  site11:x:1037:admin
  site14:x:1040:admin
  site16:x:1042:admin
  site20:x:1046:admin
  site27:x:1053:admin

In that case delete anything below the 'admin-users:x:1001:' line and you are good to go for another import attempt.

===== Technical Details =====

This [[http://devel.blueonyx.it/trac/browser/BlueOnyx/5209R/utils/blueonyx-gmu|code is pretty insane]]. No way around it:

  * [[http://devel.blueonyx.it/trac/browser/BlueOnyx/5209R/utils/blueonyx-gmu/src/blueonyx-gmu/blueonyx-gmu/verio-to-blueonyx-exporter.pl|Exporter]]
  * [[http://devel.blueonyx.it/trac/browser/BlueOnyx/5209R/utils/blueonyx-gmu/src/blueonyx-gmu/blueonyx-gmu/blueonyx-import.pl|Importer]]

Maybe it's my Swansong or my 9th Symphony. Time will tell. What it does on a technical level is this:

====The Exporter====

The current exporter is custom tailored to allow exports on Verio servers. Verio had a pretty antique hosting solution with control panel. Which did (does) run on FreeBSD 6.4 (EOL since 2010!) and CentOS 4.9 (EOL since 2012). The underlying code base of the control panel wasn't that bad, but it's clear they must have lost interest (or the know-how) to adequately maintain it. Finally they announced the EOL of it for September 2016 [[https://www.verio.eu/en/end-of-business.html|Verio CP EOL]].

Of course there is no supported way to migrate off that niche-product and this brought one of our strongest BlueOnyx supporters into a bit of a predicament, as he had a few legacy boxes on this legacy solution. Preferably he'd like to have them moved to BlueOnyx. Doing that by hand is good for a few boxes, but then it gets repetitive and boring. Hence I set off to this scripting effort.

The Verio Control Panel is "user centric". Means: You create a user first. That user can be a "domain admin". In that case he might be able to create virtual sites (one or more). Which all have their document root in subdirectories of this "domain admin".

Additional users (who aren't "domain admin") can have privileges for email, webmail, FTP and such. They can be members of a domain group, but in any case they all have their own home directory.

Regardless if the used OS is FreeBSD or CentOS: The Verio CP uses Apache, Sendmail, Procmail, Fetchmail, /etc/passwd and /etc/group. So there are no surprises. Some boxes use the mbox format, others Maildir. There is an XML file (cpx.xml) which has **some** useful info about Vsites, Users and their permissions. But other than that all config options are stored in the config files of the respective services and not in some kind of backend such as our CODB.

===Data Harvesting:===

So the best way to extract all relevant config options is to parse cpx.xml and see what we've got there. Which isn't much. 

The info about Vsites I harvested directly from httpd.conf via the Perl module [[http://search.cpan.org/~nwiger/Apache-ConfigFile-1.18/ConfigFile.pm|Apache::ConfigFile]]. That allowed me to build a Hash with all relevant info about all Vsites. For this I directly formatted the Hash keys and structure in a way that would make an import to BlueOnyx easy. The Hash keys for example already have the same key names as the CODB database fields that we want to CREATE or SET.

The info about Users is gathered from both cpx.xml and /etc/passwd parsing via [[http://search.cpan.org/~ssnodgra/Unix-ConfigFile-0.06/PasswdFile.pm|Unix::PasswdFile]] and bringing the data from both together in a sensible way.

The email aliases and server aliases? [[http://search.cpan.org/~pmqs/DB_File-1.835/DB_File.pm|DB_File]] to the rescue. With that I directly parse virtusertable.db and aliases.db to get the email configuration of users and sites and also forwards to multiple accounts or remote destinations. Throw in parsing of .procmailrc to get info about Fowarders and auto-reponders. This info complements the Users 'Email' namespace in the User's Hash.

A lot of sense checks and parsing has to be done with all harvested data. Especially the gathering of email aliases, email server aliases and forwarders turned out to be pretty complex.

For the TarBall generation I had planned to rely on [[http://search.cpan.org/~bingos/Archive-Tar-2.04/lib/Archive/Tar.pm|Archive::Tar]], but that turned out to be too slow and/or too memory hungry. So that was just done via 'tar' and 'gzip2'.

The XML file that contains all configurational options? As I didn't need anything fance the regular [[http://search.cpan.org/~grantm/XML-Simple-2.20/lib/XML/Simple.pm|XML::Simple]] was good enough.

====The Importer====

The importer is supposed to be the "receiving end" of dumps from various export platforms. So it needed to be general enough to handle all eventualities.

It's only half as complex as the exporter, as the data will have to conform to certain minimum requirements in regards to what is in the XML and the TarBalls. From there it's just the relatively simple task of reading the exported Hashes out of the XML and turning them into CCEd CREATE and SET transactions that create the desired objects and populate them with the correct data.

The only big complexity was to identify anything that could go wrong during an import and to automatically correct these problems instead of throwing an error message. 

Example: If you try to create a user that has a leading number in his username? BlueOnyx won't allow that. Hence our exporter needed the logic to check for this and to rename the user if need be (and to give him an alias of the old username). Or speaking of aliases: On the Verio CP boxes "domain admins" had the ability to create their own email aliases. That sometimes created duplicates and you ended up with two users on the same domain with the same alias. Either the importer or the exporter needed to be clever enough to not allow that. Or the import would fail.

Typically the importer has these corrective measures in place which sanitize data and prevent failures during CREATE and SET transactions.

In the close future the API of the expected import format for imports via GMU will be documented in the Wiki.
wiki.blueonyx.it

User Tools

Site Tools

Page Tools
