All websites are set up, but the https sites don't seem to be working properly. All seem to be showing bilderna.nu.

Ok, fixed it. I had to add NameVirtualHost *:443 in the SSL section of /etc/apache2/ports.conf

I've commented out the Kerberos authentication in some places - I'll have to go back and fix that, and think about how to do authentication in the future.

Continuing with websites.

I need to install the apache connector to forward servlet calls to Tomcat.

Also enabled ssl.

Did this:

sudo apt-get install libapache2-mod-jk
sudo a2enmod ssl
sudo a2enmod rewrite
sudo a2enmod dav dav_fs

Copied SSL certificates from sol to /etc/apache2/ssl.key and /etc/apache2/ssl.crt.
Copied Certificate Authority and other certs from sol's /etc/mail/certs to newly created directory /etc/certs on cumulus.

To install subversion, and the webdav connection to subversion, I did this:

sudo apt-get install subversion libapache2-svn

Copied my svn repository from sol:/usr/local/src/svn to same dir on cumulus.

Hm, svn doesn't seem to work quite yet. Or at least something is strange about the virtual web hosts...

MyDNS strangeness:

erl@cumulus:~/src/mydns-1.2.8$ dig www.voxi.com

; <<>> DiG 9.5.1-P2 <<>> www.voxi.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15541 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: Messages has 422 extra bytes at end ;; QUESTION SECTION: ;www.voxi.com. IN A ;; ANSWER SECTION: www.voxi.com. 66720 IN CNAME www.lewin.nu. www.lewin.nu. 62093 IN CNAME sol.lewin.nu. sol.lewin.nu. 62093 IN A 83.227.241.4 ;; Query time: 15 msec ;; SERVER: 172.16.1.21#53(172.16.1.21) erl@cumulus:~/src/mydns-1.2.8$ dig www.lewin.nu ; <<>> DiG 9.5.1-P2 <<>> www.lewin.nu
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3984 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1 ;; QUESTION SECTION: ;www.lewin.nu. IN A ;; ANSWER SECTION: www.lewin.nu. 86400 IN A 172.16.1.21 ;; AUTHORITY SECTION: lewin.nu. 86400 IN NS ns.lewin.nu. ;; ADDITIONAL SECTION: ns.lewin.nu. 86400 IN A 172.16.1.21 ;; Query time: 5 msec ;; SERVER: 172.16.1.21#53(172.16.1.21)

So, the first query tells me that www.voxi.com is CNAME www.lewin.nu which is CNAME sol.lewin.nu, while the second query says that www.lewin.nu is IN A 172.16.1.21. How can the same server give me two different answers for www.lewin.nu?

(while rebuilding mydns with debug enabled, installed package manpages-dev)

After debug-compiling mydns, it looks like the above query results in a recursive query, which in turn asks the bredbandsbolaget DNS server.

Found the problem - the www.voxi.com domain was not marked as 'active' in mydns' database. We should de-activate it when the new internet DNS server is online.
I'll continue debugging this tomorrow.

Web sites.

It seems like the place for web sites under Ubuntu is /var/www.
Will copy my old /srv/www/htdocs subdirectories there.

Work in progress.

Started looking at E-mail.

Installed exim4. Specified server type as 'handling incoming mail, outgoing mail by smarthost.'.

Setup virtual domains by following instructions at: http://www.debuntu.org/2006/05/17/52-how-to-exim4-virtual-host-on-debian-etch.

I think things work, but it's getting hard to test without having the server in its final extranet configuration.

Will later work on reading the exim virtual domains from the MySQL database.

Played around with Squeezeboxserver Music server on the new server.

Had some problems getting Swedish radio P1 to stream properly (it's in WMA format), worked after installing AlienBBC plugin, mplayer and lame, and specifying mms://wm-live.sr.se/sr-p1-high as the URL. Not sure what did it. Also added the following lines to /etc/squeezeboxserver/convert.conf:

wma wav * *
[mplayer] -really-quiet -vc null -vo null -cache 64 -af volume=0,resample=44100:0:1,channels=2 -ao pcm:nowaveheader:file=/dev/fd/4 $FILE$ 4>&1 1>/dev/null

wma mp3 * *
[mplayer] -really-quiet -vc null -vo null -cache 64 -af volume=0,resample=44100:0:1,channels=2 -ao pcm:nowaveheader:file=/dev/fd/4 $FILE$ 4>&1 1>/dev/null | [lame] --silent -q $QUALITY$ $RESAMPLE$ -v $BITRATE$ - -

Also downloaded Squeezeslave (from http://downloads.sourceforge.net/project/softsqueeze/squeezeslave/squeezeslave-0.9/squeezeslave-0.9-65-lnx26-alsa-display-i686.tar.gz?use_mirror=sunet). This is a command line software emulation of a Slim Devices music player. I intend to use it to play music in my living room, and control it from the web interface.

Did:

sudo cp squeezeslave /usr/local/bin

Created a startup script by modifying the one for mydns.

Installed it with:

update-rc.d squeezeslave defaults

Also made symbolic links from /media/music to my music library.

Found a bug in MyDNS which causes it to ignore the port number in the database specification.

Attached a patch to fix the bug here: http://bugs.mydns-ng.com/view.php?id=35

Installed a patched version on Cumulus (but not Cirrus yet).

Will have to redo the MyDNS imports to make sure everything goes to the right database.

Noted the ubuntu command update-rc.d which can be used when installing init.d scripts, like:

update-rc.d mydns defaults

Started copying files from my afs server to the regular filesystem on cumulus.

eth0 will be LAN, and eth1 Internet interfaces.

Started setting up two MyDNS servers. Made the /etc/init.d/mydns into two files /etc/init.d/mydns-lan and /etc/init.d/mydns-internet. Made them use different conf files, /etc/mydns-lan.conf and /etc/mydns-internet.conf. The conf files access the shared database (port 3306) or my database (port 3307).

Next task is to import the external DNS info into the right MyDNS database. I'll try to pull the info from Mårten's DNS server so I don't get the intranet version.

Building mydns on Cirrus:

wget http://downloads.sourceforge.net/project/mydns-ng/mydns/1.2.8.27/mydns-1.2.8.27.tar.gz?use_mirror=sunet
tar -xf mydns-1.2.8.27.tar.gz
cd mydns-1.2.8
sudo apt-get install g++ gcc libmysqlclient15-dev
./configure
nice make -j4
sudo make install

Then finally ran this on cumulus:

sudo mydnsimport --axfr=www.stenius.org --conf=/etc/mydns-internet.conf --replace --verbose lewin.nu

Seems to have worked. Do the same for the other zones, but pulled some of them from ns1.lewin.nu. Then import the intranet zones:

sudo mydnsimport --axfr=sol --conf=/etc/mydns-lan.conf --replace --verbose lewin.nu

In theory, all the DNS data should be in the databases, and replicated to cirrus. Checked it, looks good.

Next step is to try to serve lewin.nu intranet DNS from the new server.

Having problems specifying port for mydns server. Will continue with this later.

Trying to install openafs client on cumulus to copy data from my OpenAFS servers easily.

Did:

apt-get install openafs-client
module-assistant auto-install openafs
/etc/init.d/openafs-client force-start
apt-get install heimdal-clients

Works.

Finally think I got the replication working by adding:

max_allowed_packet = 200M

In exactly the right places in /etc/mysql/my.cnf (for every mysqld, for mysql, but not in the [client] section), on both the master and the slave.

The replication is still running, but I think it will work now.

mysqldump had stopped after 43 minutes with error:

mysqldump: Error 2020: Got packet bigger than 'max_allowed_packet' bytes when dumping table `Digitalization` at row: 3656

Updated max_allowed_packet in /etc/mysql/my.cnf to 200M, restarted copying at 15:15.

Setup phpMyAdmin on Cirrus to support the multiple databases.

Copied /etc/phpmyadmin/config-db.php to config-db1.php to config-db4.php.

Added fields so they look like:

## worry about manual changes being silently discarded. *however*,
## you'll probably also want to edit the configuration file mentioned
## above too.
##
$dbuser='phpmyadmin';
$dbpass='';
$basepath='';
$dbname='phpmyadmin';
$dbverbose='Erls Database';
$dbserver='';
$dbport='';
$dbsocket='/var/run/mysqld/mysqld2.sock';
$dbtype='mysql';

Modified central part of /etc/phpmyadmin/config.php.inc to look like:

/**
* Server(s) configuration
*/
for( $i = 1; $i < 5; $i++ )
{
// The $cfg['Servers'] array starts with $cfg['Servers'][1]. Do not use $cfg['Servers'][0].
// You can disable a server config entry by setting host to ''.
// $i++;

/* Read configuration from dbconfig-common */
require('/etc/phpmyadmin/config-db'.$i.'.php');

/* Configure according to dbconfig-common if enabled */
if (!empty($dbname)) {
/* Authentication type */
$cfg['Servers'][$i]['auth_type'] = 'cookie';
/* Server parameters */
if (empty($dbserver)) $dbserver = 'localhost';
$cfg['Servers'][$i]['host'] = $dbserver;

if (!empty($dbport)) {
$cfg['Servers'][$i]['connect_type'] = 'tcp';
$cfg['Servers'][$i]['port'] = $dbport;
}
if (!empty($dbsocket)) {
$cfg['Servers'][$i]['connect_type'] = 'socket';
$cfg['Servers'][$i]['socket'] = $dbsocket;
}
# The text name of the server to be shown in the login menu
if (!empty( $dbverbose ) )
$cfg['Servers'][$i]['verbose'] = $dbverbose;
//$cfg['Servers'][$i]['compress'] = false;
/* Select mysqli if your server has it */
$cfg['Servers'][$i]['extension'] = 'mysqli';
/* Optional: User for advanced features */
$cfg['Servers'][$i]['controluser'] = $dbuser;
$cfg['Servers'][$i]['controlpass'] = $dbpass;
/* Optional: Advanced phpMyAdmin features */
$cfg['Servers'][$i]['pmadb'] = $dbname;
$cfg['Servers'][$i]['bookmarktable'] = 'pma_bookmark';
$cfg['Servers'][$i]['relation'] = 'pma_relation';
$cfg['Servers'][$i]['table_info'] = 'pma_table_info';
$cfg['Servers'][$i]['table_coords'] = 'pma_table_coords';
$cfg['Servers'][$i]['pdf_pages'] = 'pma_pdf_pages';
$cfg['Servers'][$i]['column_info'] = 'pma_column_info';
$cfg['Servers'][$i]['history'] = 'pma_history';
$cfg['Servers'][$i]['designer_coords'] = 'pma_designer_coords';

/* Advance to next server for rest of config */
/* Now done in for loop */
// $i++;
}
}

Seems to work!

Started copying my databases to mysql2, making sure it relays to cirrus.

There was some problem at start where I had to do "STOP SLAVE" and then "START SLAVE" again on cirrus for the data to start being relayed. Strange, we'll have to keep an eye on it.

I copy data from quetzal to cumulus like this:

erl@cumulus:/etc/mysql$ mysqldump -h quetzal.bilderna.nu -P 3307 --extended-insert --databases Languages --password=*** | mysql2 --password=***

(Multiple databases can be listed after 'Languages' with spaces in between.

2.7 GB used on cumulus after the first database server has been copied.

Will now copy the bigger database with the photos.

I observed the slave being at most 1114 seconds after the master.

We need a system for removing master log files after they have been replicated to all slaves. Replication logs seem to be trimmed automatically.

I should time transferring the biggest database, which is ~25GB (My photos), just for curiosity
mysqld process on cirrus is running at 95-108 % CPU (due to Hyperthreading) and computer is reported to be ~50% idle. Doing SSL encryption of all the data is what I guess is taking up the CPU time.

Turned off the regular logging, started the biggest transfer, timing it.

Sigh. Windows update rebooted the PC I was running the stuff from, and now I have to restart. I'm glad I have this blog with the commands I was running! Started at about 11:30.

MySQL and SSL, continued

Barked up some wrong trees, but finally got it to work. I'd made some trivial errors specifying the wrong filename as --ssl-ca=... parameter to mysql, and also the permissions on the certificate file for the mysqld slave process on cirrus was wrong. But now it works for server 1!

Next step: enable slave for server 2, copy my data from quetzal to server 2 master on cumulus, make sure it ends up on cirrus.

Manually added port forwarding of ports 3307, 3308, 3309 from sol to cumulus, these settings won't survive a reboot.

Steps to enable replication:

On master:

mysql> flush tables with read lock;
Query OK, 0 rows affected (0.00 sec)

mysql> show master status;
+-------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| mysql2-bin.000005 | 98 | | |
+-------------------+----------+--------------+------------------+
1 row in set (0.00 sec)
mysql> flush tables with read lock;
Query OK, 0 rows affected (0.00 sec)

mysql> show master status;
+-------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+-------------------+----------+--------------+------------------+
| mysql2-bin.000005 | 98 | | |
+-------------------+----------+--------------+------------------+
1 row in set (0.00 sec)

On slave:

mysql> CHANGE MASTER TO MASTER_HOST='www.lewin.nu', MASTER_USER='replicationSlave', MASTER_PASSWORD='****', MASTER_LOG_FILE='mysql2-bin.000005', MASTER_LOG_POS=98, MASTER_SSL=1, MASTER_SSL_CA='/etc/mysql/mysql-ca-cert.pem', MASTER_PORT=3307;
Query OK, 0 rows affected (0.01 sec)

For some reason I got errors about mysql not being able to update the master.info; I had to to 'RESET SLAVE', then CHANGE MASTER TO..., and then it would work.

Then do "unlock tables" on the master, and "start slave" on the slave.

do "SHOW SLAVE STATUS\G;" on the slave, the first line should read: "Waiting for master to send event".

Next step is to copy my date from quetzal to database 2, and make sure it is replicated to Mårten. It's tens of GB I think, so it will take a while.

Bedtime now!

SSL problem.

I now have problems with SSL and mysql.

I can connect from sol to cumulus with ssl:

erl@sol:~$ mysql -h 172.16.1.21 --ssl --ssl-ca=/tmp/ca-cert.pem -P 3306 -p

But I can't do the same thing from cirrus via port 3306 forwarded from sol to cumulus:

erl@cirrus:~$ mysql --ssl --ssl-ca=ca-cert.pem -h www.lewin.nu -p -P 3306
Enter password:
ERROR 2026 (HY000): SSL connection error

Does the problem have to do with port forwarding?

From symphony.stenius.org, I can:

[erl@symphony erl]$ mysql -h www.lewin.nu -p
Enter password:
ERROR 1251: Client does not support authentication protocol requested by server; consider upgrading MySQL client

Which indicates that the port forwarding works, but that symphony doesn't have an ssl enabled mysql client.

Doing the same thing on cirrus:

erl@cirrus:~$ mysql -h www.lewin.nu -p
Enter password:
ERROR 1045 (28000): Access denied for user 'erl'@'1-1-6-5a.o.sth.bostream.se' (using password: YES)

Looks like the port forwarding works to me. How to debug the ssl?

Found this on debugging ssl connections:

http://www.cyberciti.biz/tips/debugging-ssl-communications-from-unix-shell-prompt.html

So I try this:

erl@cirrus:~$ openssl s_client -ssl2 -connect sol.lewin.nu:3306
CONNECTED(00000003)
17539:error:1407F0E5:SSL routines:SSL2_WRITE:ssl handshake failure:s2_pkt.c:428:

Sigh. Stuck here.

New day - new tasks.

Now want to set up MySQL replication, at first from my server to Mårten's.
Also modifying MySQL setup to support multiple datbases.

Found this blog post which seems relevant:

http://blog.erichsen.net/2009/09/01/multiple-mysql-slave-instances-on-a-single-server/

Also found this page:

http://ubuntuforums.org/archive/index.php/t-1154296.html

Followed the instructions on the second link. Got bit by one thing: apparently Ubuntu runs apparmor by default, I needed to modify /etc/apparmor.d/usr.sbin.mysqld to contain:

/var/lib/mysql?/ r,
/var/lib/mysql?/** rwk,
/var/log/mysql/ r,
/var/log/mysql/** rw,
/var/run/mysqld/mysqld?.pid w,
/var/run/mysqld/mysqld?.sock w,

I now have four mysql servers on cumulus.

Repeated instructions on cirrus. Only enabled logging on master servers.

Next step is to setup cirrus to replicate databases 1 and 2 from cumulus.

Continuing with new server.

Added port forwarding from old server to new server's mysql:

/sbin/iptables -t nat -A PREROUTING -p tcp -i eth-ext -d 83.227.241.4 --dport 3306 -j DNAT --to 172.16.1.21:3306
/sbin/iptables -A FORWARD -p tcp -i eth-ext --dport 3306 -j ACCEPT

Now I need to change the new server's mysql server to bind to all ip addresses. Comment out the line with bind-address in /etc/mysql/my.cnf

Followed instructions here to create SSL certificates for MySQL:

http://dev.mysql.com/doc/refman/5.0/en/secure-create-certs.html

Copy the certificates mentioned in /etc/mysql/my.cnf to /etc/mysql

Uncomment the ssl lines in /etc/mysql/my.cnf

Restart mysql.

Added privileges to require SSL for external MySQL connections. For my user, added:

erl@%
erl@localhost
erl@172.16.%.%

For the first one, in a text mysql session, do something like:

show grants for 'erl';

This will show something like:

+-------------------------------------------------------------------------------------------------------------------------------------------+
| Grants for erl@% |
+-------------------------------------------------------------------------------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'erl'@'%' IDENTIFIED BY PASSWORD '*XXX' WITH GRANT OPTION |
+-------------------------------------------------------------------------------------------------------------------------------------------+

Modify the above and add REQUIRE SSL before WITH GRANT OPTION:

GRANT ALL PRIVILEGES ON * . * TO 'erl'@'%' IDENTIFIED BY PASSWORD '*XXX' REQUIRE SSL WITH GRANT OPTION MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0 ;

The MySQL ca-cert.pem file must then be copied to the client (!), in order for mysql to allow the connection. The mysql client program must be started like:

mysql --ssl --ssl-ca=ca-cert.pem -h www.lewin.nu -p

Next set is to set up replication, so that Mårten's server's main MySQL database is a slave to Erland's.



Installing MyDNS under Ubuntu 9.04

I want to try running MyDNS on my new server - a DNS server with its data in a MySQL database.

At first I started getting MyDNS, but it seems like the latest release is called MyDNS NG, at https://sourceforge.net/projects/mydns-ng/

There seems to be no package for Ubuntu 9.04.

I downloaded the source from http://mydns.bboy.net/download/mydns-1.1.0.tar.bz2
I needed to install gcc: apt-get install gcc
I needed to install a mysql client library: apt-get install mysql-client
Also needed: apt-get install make libmysqlclient15-dev
I unpacked the source, and could now run ./configure successfully

Basically followed instructions at: http://www.howtoforge.com/installing-mydns-ng-and-mydnsconfig-3-on-debian-lenny

(make, make install, create startup script, sudo update-rc.d mydns defaults)

downloaded MyDNSConfig from https://sourceforge.net/projects/mydnsconfig/

sudo apt-get install php5-cli vlogger

Hm, mydnsconfig doesn't seem to work 100% (Doesn't show DNS rules), will not use it for now.

PhpMyAdmin installation on new server

I've started installing my new server 'cumulus' based on an Intel D945GSEJT motherboard with an Intel Atom N270 processor and a Western Digital Caviar Green WD15EADS 1.5TB hard drive. I'm using Ubuntu 9.04 Server.

I could easily mount the motherboard in an old PC case, the holes and spacers fit.

The IDE/PATA connector on the motherboard is a laptop standard, and will require an adaptor to connect a standard IDE connector.

Now installing phpmyadmin, did:

sudo apt-get install phpmyadmin

Added:

Include /etc/phpmyadmin/apache.conf

at the end of /etc/apache2/apache2.conf according to https://help.ubuntu.com/community/phpMyAdmin

Restarted apache with /etc/init.d/apache2 restart

I can now access phpMyAdmin at http://computer.../phpmyadmin

Problems with Berkeley DB, Apache apr-util, and Subversion

I'm trying to build a recent version of subversion on my server, and ran into the following problem:

I installed Berkeley db-4.7.25, using only --prefix=/usr as a configure parameter, built and installed. No problem.

I then built and installed apache's apr-util, version 1.3.7. I used 'configure --prefix=/usr --with-apr=/usr --with-berkeley-db=/usr as parameters.

Then I tried to configure subversion, version 1.6.3, with configure parameters'--prefix=/usr' '--with-ssl' '--with-apxs=/usr/apache2/bin/apxs' '--enable-shared' '--with-apr-util=/usr/bin/apu-1-config'.

However, I go the following error, despite trying various changes:

configure: WARNING: we have configured without BDB filesystem support


You don't seem to have Berkeley DB version 4.0.14 or newer
installed and linked to APR-UTIL. We have created Makefiles which
will build without the Berkeley DB back-end; your repositories will
use FSFS as the default back-end. You can find the latest version of
Berkeley DB here:
http://www.oracle.com/technology/software/products/berkeley-db/index.html

The problem seems to stem from the fact that apu-1-config --libs doesn't list "-ldb-4.7" as a library to be linked with. I did a manual hack in the configure file of subversion to add this. At around line 23435:

LIBS="`$apu_config --ldflags --link-ld` $SVN_DB_LIBS $LIBS -ldb-4.7"

Posting this in case it helps anybody else. By the way, I think the --link-ld flag above is something I added, but that probably wasn't neccessary in the end.

Sigh. Configuring worked, building now, hope it works.