Author Topic: Latest update Network Configurator borks server  (Read 2725 times)

Escorpiom

  • Zen Hero
  • *****
  • Posts: 897
  • Karma: +25/-1
    • View Profile
Latest update Network Configurator borks server
« on: November 10, 2012, 10:06:22 pm »
This morning I noticed a component update to the network configurator (Zentyal version 2.2.7) and installed it.
After saving the changes, there was no internet anymore, not on the server and not on the clients.

I restarted the server and it was obvious the firewall and network were not starting, as observed from the list of modules.
There are 4 network cards in the server, 2 external and two internal. The connection type is PPPoE.
At this moment I have started the network and firewall from the terminal and at least it enables me to post this message, but the network is still down for other clients on the network.

Please review this update because something went terribly wrong, the server is unusable at the moment. If you need more info please let me know.

Edit: The boot log, observe network and firewall not starting:
Code: [Select]
* Starting Zarafa server: zarafa-server [240G [234G[ OK ]
 * Starting Zarafa spooler: zarafa-spooler       [240G [234G[ OK ]
 * Starting Zentyal module: network              [80G [74G [31mfail [39;49m]
 * Starting Zentyal module: firewall               [80G [74G[31mfail [39;49m]
 * Starting Zentyal module: antivirus            [80G [74G[ OK ]
 * Starting Zentyal module: audit                 [80G [74G[ OK ]
 * Starting Zentyal module: ca                     [80G [74G[ OK ]
 * Starting Zentyal module: dhcp                 [80G [74G[ OK ]
 * Starting Zentyal module: dns                  [80G [74G[ OK ]
 * Starting Zentyal module: ebackup             [80G [74G[ OK ]
 * Starting Zentyal module: events               [80G [74G[ OK ]
 * Starting Zentyal module: ids                    [80G [74G[ OK ]
 * Starting Zentyal module: ipsec                [80G [74G[ OK ]
 * Starting Zentyal module: logs                 [80G [74G[ OK ]
 * Starting Zentyal module: mailfilter            [80G [74G[ OK ]
 * Starting Zentyal module: monitor             [80G [74G[ OK ]
 * Starting Zentyal module: ntp                  [80G [74G[ OK ]
 * Starting Zentyal module: openvpn           [80G [74G[ OK ]
 * Starting Zentyal module: pptp                [80G [74G[ OK ]
 * Starting Zentyal module: printers            [80G [74G[ OK ]
ping: unknown host vpn1.cloud.zentyal.com
ping: unknown host vpn1.cloud.zentyal.com
ping: unknown host vpn1.cloud.zentyal.com
ping: unknown host vpn1.cloud.zentyal.com
ping: unknown host vpn1.cloud.zentyal.com
ping: unknown host vpn1.cloud.zentyal.com
ping: unknown host vpn1.cloud.zentyal.com
ping: unknown host vpn1.cloud.zentyal.com
ping: unknown host vpn1.cloud.zentyal.com
ping: unknown host vpn1.cloud.zentyal.com
ping: unknown host vpn1.cloud.zentyal.com
ping: unknown host vpn1.cloud.zentyal.com
 * Starting Zentyal module: remoteservices        [80G 74G[ OK ]
 * Starting Zentyal module: trafficshaping          [80G

This is a piece of the Zentyal log file when booting:
Code: [Select]
Stack:
  [/usr/share/perl5/HTML/Mason/Request.pm:127]
  [/usr/share/perl5/HTML/Mason/Request.pm:525]
2012/11/10 15:48:14 INFO> Redis.pm:766 EBox::Config::Redis::_initRedis - Starting redis server
2012/11/10 15:48:20 INFO> Base.pm:250 EBox::Module::Base::__ANON__ - Saving config for module: network
2012/11/10 15:48:25 ERROR> Ldap.pm:1103 EBox::Ldap::safeConnect - Couldn't connect to LDAP server ldapi://%2fvar%2frun%2fslapd%2fldapi, retrying
2012/11/10 15:48:30 ERROR> Ldap.pm:1103 EBox::Ldap::safeConnect - Couldn't connect to LDAP server ldapi://%2fvar%2frun%2fslapd%2fldapi, retrying
2012/11/10 15:48:31 INFO> Ldap.pm:1112 EBox::Ldap::safeConnect - LDAP reconnect successful
2012/11/10 15:48:32 ERROR> ppp-set-iface.pl:40 main::__ANON__ - Call to setRealPPPIface for eth3 failed
2012/11/10 15:48:33 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: firewall
2012/11/10 15:48:35 DEBUG> Base.pm:241 EBox::RemoteServices::Base::_queryServicesNameserver - Server vpn1.cloud.zentyal.com not found via DNS server ns.cloud.zentyal.com,127.0.0.1,8.8.4.4,8.8.8.8. Reason: SERVFAIL
2012/11/10 15:48:35 ERROR> Iptables.pm:373 EBox::Iptables::__ANON__ - Cannot contact Zentyal Cloud: Server vpn1.cloud.zentyal.com not found via DNS server ns.cloud.zentyal.com,127.0.0.1,8.8.4.4,8.8.8.8. Reason: SERVFAIL
2012/11/10 15:48:38 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: network
2012/11/10 15:48:38 ERROR> Service.pm:721 EBox::Module::Service::__ANON__ - Error restarting service: Can't call method "address" on an undefined value at /usr/share/perl5/EBox/Network.pm line 4121.
2012/11/10 15:48:38 ERROR> Lock.pm:31 EBox::Util::Lock::lock - Could not get lock for service: firewall
2012/11/10 15:48:38 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: antivirus
2012/11/10 15:48:44 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: audit
2012/11/10 15:48:44 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: ca
2012/11/10 15:48:44 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: dhcp
2012/11/10 15:48:45 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: dns
2012/11/10 15:48:46 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: ebackup
2012/11/10 15:48:46 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: events
2012/11/10 15:48:47 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: ids
2012/11/10 15:48:47 INFO> EventDaemon.pm:307 EBox::EventDaemon::_loadModules - EBox::Event::Watcher::Updates loaded from registeredEvents
2012/11/10 15:48:47 INFO> EventDaemon.pm:307 EBox::EventDaemon::_loadModules - EBox::Event::Watcher::EBackup loaded from registeredEvents
2012/11/10 15:48:47 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: ipsec
2012/11/10 15:48:47 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: logs
2012/11/10 15:48:47 INFO> EventDaemon.pm:307 EBox::EventDaemon::_loadModules - EBox::Event::Dispatcher::ControlCenter loaded from registeredDispatchers
2012/11/10 15:48:47 INFO> EventDaemon.pm:307 EBox::EventDaemon::_loadModules - EBox::Event::Dispatcher::Log loaded from registeredDispatchers
2012/11/10 15:48:47 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: mailfilter
2012/11/10 15:48:49 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: monitor
2012/11/10 15:48:50 WARN> Monitor.pm:662 EBox::Monitor::_setThresholdConf - No threshold configuration is saved since monitor watcher or events module are not enabled
2012/11/10 15:48:50 DEBUG> Base.pm:241 EBox::RemoteServices::Base::_queryServicesNameserver - Server mon.internal.cloud.zentyal.com not found via DNS server 10.200.0.4. Reason: Send error: Network is unreachable
2012/11/10 15:48:52 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: ntp
2012/11/10 15:48:53 ERROR> Sudo.pm:213 EBox::Sudo::_rootError - root command set -e
/sbin/iptables -t filter -A iexternalmodules -i tap0 -p udp --destination-port 520 -j ACCEPT
/sbin/iptables -t filter -A imodules -i tap0 -p udp --destination-port 520 -j ACCEPT
/sbin/iptables -t filter -A omodules -o tap0 -p udp --destination-port 520 -j ACCEPT
/sbin/iptables -t filter -A omodules --protocol tcp --destination vpn1.cloud.zentyal.com --destination-port 1194 -j ACCEPT
/sbin/iptables -t filter -A omodules --protocol tcp  --destination-port 80 -j ACCEPT failed.
Error output: iptables v1.4.4: host/network `vpn1.cloud.zentyal.com' not found
 Try `iptables -h' or 'iptables --help' for more information.

Command output: .
Exit value: 2
2012/11/10 15:48:53 ERROR> Iptables.pm:687 EBox::Iptables::__ANON__ - Error executing firewall rules for module openvpn
2012/11/10 15:48:54 ERROR> Sudo.pm:213 EBox::Sudo::_rootError - root command /usr/sbin/ntpdate 0.pool.ntp.org failed.
Error output: Name server cannot be used, exiting10 Nov 15:48:54 ntpdate[6555]: name server cannot be used, reason: Temporary failure in name resolution
 

Command output: .
Exit value: 1
2012/11/10 15:48:54 WARN> NTP.pm:154 EBox::NTP::__ANON__ - Couldn't execute ntpdate 0.pool.ntp.org
2012/11/10 15:48:54 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: openvpn
2012/11/10 15:48:54 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: pptp
2012/11/10 15:48:54 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: printers
2012/11/10 15:48:56 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: remoteservices
2012/11/10 15:48:56 DEBUG> Base.pm:241 EBox::RemoteServices::Base::_queryServicesNameserver - Server backup.internal.cloud.zentyal.com not found via DNS server 10.200.0.4. Reason: Send error: Network is unreachable
2012/11/10 15:48:56 ERROR> RemoteServices.pm:1590 EBox::RemoteServices::__ANON__ - Cannot contact to Zentyal Cloud: Server backup.internal.cloud.zentyal.com not found via DNS server 10.200.0.4. Reason: Send error: Network is unreachable
2012/11/10 15:48:57 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: trafficshaping
2012/11/10 15:48:57 DEBUG> TrafficShaping.pm:1104 EBox::TrafficShaping::_checkInterface - Interface ppp0 does not exist.
2012/11/10 15:48:57 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: users
2012/11/10 15:48:58 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: virt
2012/11/10 15:48:58 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: webserver
2012/11/10 15:48:59 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: asterisk
2012/11/10 15:49:00 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: bwmonitor
2012/11/10 15:49:00 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: captiveportal
2012/11/10 15:49:00 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: ftp
2012/11/10 15:49:00 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: jabber
2012/11/10 15:49:01 INFO> CaptiveDaemon.pm:233 main:: - Starting Captive Portal Daemon
2012/11/10 15:49:07 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: mail
2012/11/10 15:49:10 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: radius
2012/11/10 15:49:10 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: samba
2012/11/10 15:49:12 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: squid
2012/11/10 15:49:16 INFO> Base.pm:250 EBox::Module::Base::__ANON__ - Saving config for module: network
2012/11/10 15:49:19 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: usercorner
2012/11/10 15:49:20 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: webmail
2012/11/10 15:49:22 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: zarafa
2012/11/10 15:49:23 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: firewall
2012/11/10 15:49:24 DEBUG> Base.pm:241 EBox::RemoteServices::Base::_queryServicesNameserver - Server vpn1.cloud.zentyal.com not found via DNS server ns.cloud.zentyal.com,127.0.0.1,8.8.4.4,8.8.8.8. Reason: SERVFAIL
2012/11/10 15:49:24 ERROR> Iptables.pm:373 EBox::Iptables::__ANON__ - Cannot contact Zentyal Cloud: Server vpn1.cloud.zentyal.com not found via DNS server ns.cloud.zentyal.com,127.0.0.1,8.8.4.4,8.8.8.8. Reason: SERVFAIL
2012/11/10 15:49:26 INFO> Service.pm:716 EBox::Module::Service::restartService - Restarting service for module: apache
2012/11/10 15:49:44 ERROR> Sudo.pm:213 EBox::Sudo::_rootError - root command set -e
/sbin/iptables -t filter -A iexternalmodules -i tap0 -p udp --destination-port 520 -j ACCEPT
/sbin/iptables -t filter -A imodules -i tap0 -p udp --destination-port 520 -j ACCEPT
/sbin/iptables -t filter -A omodules -o tap0 -p udp --destination-port 520 -j ACCEPT
/sbin/iptables -t filter -A omodules --protocol tcp --destination vpn1.cloud.zentyal.com --destination-port 1194 -j ACCEPT
/sbin/iptables -t filter -A omodules --protocol tcp  --destination-port 80 -j ACCEPT failed.
Error output: iptables v1.4.4: host/network `vpn1.cloud.zentyal.com' not found
 Try `iptables -h' or 'iptables --help' for more information.

Part of the installation log, the culprit is Zentyal Network module 2.2.9
Code: [Select]
2012-11-10 05:43:58> Zentyal apt-wrapper install started
2012-11-10 05:43:58> Reading package lists...
2012-11-10 05:43:58> Building dependency tree...
2012-11-10 05:43:58> Reading state information...
2012-11-10 05:43:58> The following packages will be upgraded:
2012-11-10 05:43:58>   zentyal-network
2012-11-10 05:43:59> 1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2012-11-10 05:43:59> Need to get 75.5kB of archives.
2012-11-10 05:43:59> After this operation, 0B of additional disk space will be used.
2012-11-10 05:43:59> Get:1 http://ppa.launchpad.net/zentyal/2.2/ubuntu/ lucid/main zentyal-network 2.2.9 [75.5kB]
2012-11-10 05:44:00> Fetched 75.5kB in 1s (72.9kB/s)
2012-11-10 05:44:02> (Reading database ... 139571 files and directories currently installed.)
2012-11-10 05:44:02> Preparing to replace zentyal-network 2.2.8 (using .../zentyal-network_2.2.9_all.deb) ...
2012-11-10 05:44:02> Unpacking replacement zentyal-network ...
2012-11-10 05:44:02> Setting up zentyal-network (2.2.9) ...
2012-11-10 05:44:15>  * Restarting Zentyal module: network
2012-11-10 05:44:15>    ...done.
2012-11-10 05:44:15>
2012-11-10 05:44:15> Processing triggers for zentyal-core ...
2012-11-10 05:44:16>  * Restarting Zentyal module: apache
2012-11-10 05:44:16>    ...done.
2012-11-10 05:44:18>  * Restarting Zentyal module: logs
2012-11-10 05:44:18>    ...done.
2012-11-10 05:44:20>  * Restarting Zentyal module: events
2012-11-10 05:44:20>    ...done.
2012-11-10 05:44:21> Zentyal apt-wrapper install finished

Cheers.
« Last Edit: November 11, 2012, 07:30:47 am by Escorpiom »
Marcus' Rule:
Blanks & capitals = avoid it and you'll avoid problems...

Escorpiom

  • Zen Hero
  • *****
  • Posts: 897
  • Karma: +25/-1
    • View Profile
Re: Latest update Network Configurator borks server
« Reply #1 on: November 11, 2012, 12:58:16 am »
OK I do not know -why- and -how- it happened, but I do know -what- happened.
After the Network Configurator update, all my firewall rules have disappeared, including the ones setup by Zentyal at install time.
In the packet filter screens, everything is just totally blank.
The only screen not affected is "Rules added by Zentyal services (Advanced)".

So the question is, how can I restore the default Zentyal rules?
Does anyone have some screens of the ruleset as added by Zentyal? 
I hope to get at least some functionality back. 

Cheers.
Marcus' Rule:
Blanks & capitals = avoid it and you'll avoid problems...

Sam Graf

  • Guest
Re: Latest update Network Configurator borks server
« Reply #2 on: November 11, 2012, 02:49:32 am »
I'm very sorry, I'm not at the office so cannot help. :(

Do you have an up-to-date configuration backup you could try to restore?

Escorpiom

  • Zen Hero
  • *****
  • Posts: 897
  • Karma: +25/-1
    • View Profile
Re: Latest update Network Configurator borks server
« Reply #3 on: November 11, 2012, 05:42:59 am »
Sadly not Sam, but after putting in the rules I could "guess" were online again. Yeah!
Surely there are some rules missing and perhaps something won't work because of that, but what worries me the most: What will happen when the server reboots?
Will everything get erased again?
I hope Zentyal devs can take a look at the update and what it does, took me a couple of hours to put it together again and the people here were not happy...

Cheers.
Marcus' Rule:
Blanks & capitals = avoid it and you'll avoid problems...

Sam Graf

  • Guest
Re: Latest update Network Configurator borks server
« Reply #4 on: November 11, 2012, 06:05:26 am »
Glad you were able to get everyone back on line.

I would be surprised if Zentyal deletes firewall rules on reboot. But if that's still a possibility, it'll also be a possibility if something causes the firewall module to restart.

In any case, if the update currently is a risky proposition we might see others with the same problem. Thank you for doing the troubleshooting on what went wrong. Hopefully the developers can determine the how and why.