Differences

This shows you the differences between two versions of the page.

--- info:devel:gmu [2015/11/28 08:32]
+++ info:devel:gmu [2015/12/31 10:49] (current)
@@ Line 8: / Line 8: @@
 Please note: Currently the only supported platform for exports to BlueOnyx are Verio FreeBSD and Verio CentOS-4 servers that run the Verio GUI. Support for further platforms will be added later (think Plesk & cPanel).
+Additionally: MySQL will **not** be exported or imported. You will need to move that manually.
 ===== How to export: =====
@@ Line 13: / Line 15: @@
 ==What you need:==
-  * '//root//' access to a supported export platform (Verio FreeBSD 6.4 or Verio CentOS4.9)
+  * '//root//' access to a supported export platform (Verio FreeBSD 6.4 or Verio CentOS 4.9)
   * A couple of GB of free space on that server.
@@ Line 74: / Line 76: @@
 To do the actual export run one of the two commands:
-=To store export locally:=
+**To store export locally:**
   export_box# ./verio-to-blueonyx-exporter.pl 2>&1 | tee export.log
-=To store export remotely:=
+**To store export remotely:**
 On the remote server add '/root/.ssh/id_rsa.pub' from the export server to '/root/.ssh/authorized_keys'. Then create the import directory there:
@@ Line 100: / Line 102: @@
 ==Setup of the importer:==
-The importer is named 'blueonyx-gmu' and is available in the BlueOnyx 5209R YUM repository. To install it on a 5209R run the following command:
+The RPM that contains the importer is named 'blueonyx-gmu' and is available in the BlueOnyx 5209R YUM repository. To install it on a 5209R run the following command:
   yum install blueonyx-gmu
@@ Line 135: / Line 137: @@
 | -r  | <vsites><users><resellers>     | If an import failed for any reason, then you can use '-r vsites' to remove all Vsites (and Users). If you want to only remove all Users (not System-Administrators!), then you can use '-r users'. If you use '-r resellers', then all Reseller accounts (including their Vsites and Users!) will be removed. |
 | -a  | <none>     | If specified, no Reseller accounts will be created during import. |
+| -s  | <none>     | Vsite that already exists on server doesn't get imported. |
 | -e  | <none>     | Does not perform an import. Instead it will check the integrity of the XML file and the TarBalls in the specified import directory. |
-| -q  | <none>     | Does not perform an import. If with this option, then any Vsite or User that is over quota (or at +90% disk usage) will receive enough additional disk space to end the over-quota situation. This option is used automatically at the end of an import. |
+| -q  | <none>     | Does not perform an import. If run with this option, then any Vsite or User that is over quota (or at +90% disk usage) will receive enough additional disk space to end the over-quota situation. This option is used automatically at the end of an import. |
+| -p  | <none>     | Does not perform an import. This will read the server wide PHP settings and will apply them to all Vsites. |
 | -h  | <none>     | Display the help text and exit.   |
@@ Line 193: / Line 197: @@
 ==== During Import ====
+The import utility should give you a good indication about the nature of any problem that it did run into. Also see '/var/log/messages' for any CCEd related error messages that happened during CREATE or SET transactions.
+**The most probable causes of failure during import are:**
+  * /etc/mail/aliases did contain entries that conflict with whatever was being added during User or Vsite creation. Solution: Clean up /etc/mail/aliases to remove dormant or conflicting entries.
+  * Username or Vsite name already taken. Solution: Only do an import on a 5209R that doesn't yet contain any Vsites and Users. Use the '-r' option of the importer to remove Vsites and Users before you do an import run.
+  * Email- or Web-Alias already taken. The importer takes great care to not set any aliases that are already taken by other Vsites and/or Users. If it detects such a conflict, it will import the Vsite (or User) without the conflicting alias.
+**Uncritical messages:**
+During imports of TarBalls you might see something like this:
+  tar: Ignoriere unbekanntes Schlüsselwort „SCHILY.dev“ für erweiterten Kopfteil
+  tar: Ignoriere unbekanntes Schlüsselwort „SCHILY.ino“ für erweiterten Kopfteil
+  tar: Ignoriere unbekanntes Schlüsselwort „SCHILY.nlink“ für erweiterten Kopfteil
+That can be ignored. The TarBalls will still import fine and this error message (actually it's just a 'notice' and not an error) is caused by drastically different TAR versions and/or MacOS related. Like if a user FTP'ed in stuff with his MacBook.
+=== An Import failed. Now what? ===
+Check the error message you got. Check /var/log/messages for the SET and CREATE transaction that failed. Try to run that SET or CREATE transaction manually from '/usr/sausalito/bin/cceclient' as that might give you a more detailed error message or further indications about the nature of the problem.
+It **could** also be that it was just CCEd that was a bit temperamental during the import and decided to take a dive.
+Once you solved the problem and want to try another import you need to remove all Vsites, Users and Resellers first. To do so, run these two commands:
+  /usr/sausalito/sbin/blueonyx-import.pl -r vsites
+  /usr/sausalito/sbin/blueonyx-import.pl -r resellers
+That will delete all Vsites and all Users and also all Resellers. It will *not* remove the 'admin' account or any GUI created administrative users with the System-Administrator flag set.
+Afterwards there are two places you also want to check before attempting another import. Usually the GUI takes care of removing these entries, but sometimes some junk remains around. So please run ...
+  cat /etc/mail/aliases
+This should be pretty bare. The last good lines of it should read like this:
+  # Person who should get root's mail
+  #root:          marc
+  root:           admin
+  root-admin:     admin
+If there is anything **below** that (which doesn't relate to a System-Administrator), then delete these lines.
+  cat /etc/group
+The last couple of lines should read like this:
+  sshd:x:74:
+  httpd:x:48:
+  site-adm:x:1000:
+  admin-users:x:1001:
+In that case: All good. If it looks like this (**after** you deleted all Vsites), then you have a problem:
+  sshd:x:74:
+  httpd:x:48:
+  site-adm:x:1000:
+  admin-users:x:1001:
+  site1:x:1027:admin
+  site2:x:1028:admin
+  site6:x:1032:admin
+  site11:x:1037:admin
+  site14:x:1040:admin
+  site16:x:1042:admin
+  site20:x:1046:admin
+  site27:x:1053:admin
+In that case delete anything below the 'admin-users:x:1001:' line and you are good to go for another import attempt.
+===== Technical Details =====
+This [[http://devel.blueonyx.it/trac/browser/BlueOnyx/5209R/utils/blueonyx-gmu|code is pretty insane]]. No way around it:
+  * [[http://devel.blueonyx.it/trac/browser/BlueOnyx/5209R/utils/blueonyx-gmu/src/blueonyx-gmu/blueonyx-gmu/verio-to-blueonyx-exporter.pl|Exporter]]
+  * [[http://devel.blueonyx.it/trac/browser/BlueOnyx/5209R/utils/blueonyx-gmu/src/blueonyx-gmu/blueonyx-gmu/blueonyx-import.pl|Importer]]
+Maybe it's my Swansong or my 9th Symphony. Time will tell. What it does on a technical level is this:
+====The Exporter====
+The current exporter is custom tailored to allow exports on Verio servers. Verio had a pretty antique hosting solution with control panel. Which did (does) run on FreeBSD 6.4 (EOL since 2010!) and CentOS 4.9 (EOL since 2012). The underlying code base of the control panel wasn't that bad, but it's clear they must have lost interest (or the know-how) to adequately maintain it. Finally they announced the EOL of it for September 2016 [[https://www.verio.eu/en/end-of-business.html|Verio CP EOL]].
+Of course there is no supported way to migrate off that niche-product and this brought one of our strongest BlueOnyx supporters into a bit of a predicament, as he had a few legacy boxes on this legacy solution. Preferably he'd like to have them moved to BlueOnyx. Doing that by hand is good for a few boxes, but then it gets repetitive and boring. Hence I set off to this scripting effort.
+The Verio Control Panel is "user centric". Means: You create a user first. That user can be a "domain admin". In that case he might be able to create virtual sites (one or more). Which all have their document root in subdirectories of this "domain admin".
+Additional users (who aren't "domain admin") can have privileges for email, webmail, FTP and such. They can be members of a domain group, but in any case they all have their own home directory.
+Regardless if the used OS is FreeBSD or CentOS: The Verio CP uses Apache, Sendmail, Procmail, Fetchmail, /etc/passwd and /etc/group. So there are no surprises. Some boxes use the mbox format, others Maildir. There is an XML file (cpx.xml) which has **some** useful info about Vsites, Users and their permissions. But other than that all config options are stored in the config files of the respective services and not in some kind of backend such as our CODB.
+===Data Harvesting:===
+So the best way to extract all relevant config options is to parse cpx.xml and see what we've got there. Which isn't much.
+The info about Vsites I harvested directly from httpd.conf via the Perl module [[http://search.cpan.org/~nwiger/Apache-ConfigFile-1.18/ConfigFile.pm|Apache::ConfigFile]]. That allowed me to build a Hash with all relevant info about all Vsites. For this I directly formatted the Hash keys and structure in a way that would make an import to BlueOnyx easy. The Hash keys for example already have the same key names as the CODB database fields that we want to CREATE or SET.
+The info about Users is gathered from both cpx.xml and /etc/passwd parsing via [[http://search.cpan.org/~ssnodgra/Unix-ConfigFile-0.06/PasswdFile.pm|Unix::PasswdFile]] and bringing the data from both together in a sensible way.
+The email aliases and server aliases? [[http://search.cpan.org/~pmqs/DB_File-1.835/DB_File.pm|DB_File]] to the rescue. With that I directly parse virtusertable.db and aliases.db to get the email configuration of users and sites and also forwards to multiple accounts or remote destinations. Throw in parsing of .procmailrc to get info about Fowarders and auto-reponders. This info complements the Users 'Email' namespace in the User's Hash.
+A lot of sense checks and parsing has to be done with all harvested data. Especially the gathering of email aliases, email server aliases and forwarders turned out to be pretty complex.
+For the TarBall generation I had planned to rely on [[http://search.cpan.org/~bingos/Archive-Tar-2.04/lib/Archive/Tar.pm|Archive::Tar]], but that turned out to be too slow and/or too memory hungry. So that was just done via 'tar' and 'gzip2'.
+The XML file that contains all configurational options? As I didn't need anything fance the regular [[http://search.cpan.org/~grantm/XML-Simple-2.20/lib/XML/Simple.pm|XML::Simple]] was good enough.
+====The Importer====
+The importer is supposed to be the "receiving end" of dumps from various export platforms. So it needed to be general enough to handle all eventualities.
+It's only half as complex as the exporter, as the data will have to conform to certain minimum requirements in regards to what is in the XML and the TarBalls. From there it's just the relatively simple task of reading the exported Hashes out of the XML and turning them into CCEd CREATE and SET transactions that create the desired objects and populate them with the correct data.
+The only big complexity was to identify anything that could go wrong during an import and to automatically correct these problems instead of throwing an error message.
+Example: If you try to create a user that has a leading number in his username? BlueOnyx won't allow that. Hence our exporter needed the logic to check for this and to rename the user if need be (and to give him an alias of the old username). Or speaking of aliases: On the Verio CP boxes "domain admins" had the ability to create their own email aliases. That sometimes created duplicates and you ended up with two users on the same domain with the same alias. Either the importer or the exporter needed to be clever enough to not allow that. Or the import would fail.
+Typically the importer has these corrective measures in place which sanitize data and prevent failures during CREATE and SET transactions.
+In the close future the API of the expected import format for imports via GMU will be documented in the Wiki.

wiki.blueonyx.it

User Tools

Site Tools

Differences

Page Tools