[SOLVED] Zentyal stop working every morning

[Daniel Marques]

My office use zentyal for few mounth now without troubleshoting.
But from a few weeks, every morning between 7h and 9h, Zentyal stop working. DHCP no more deliver IP adresses, Can't receive or send Email, can't access zentyal by ssh or browser.

Rebooting zentyal fixe the problem until next morning.

I have spent a lot of time reading syslog file without any result, but i don't really know what to look for.
Can somebody help me?

[christian]

1 - Does Zentyal restart / reboot ?
2 - Look at syslog and also /var/log/zentyal/zentyal.log
3 - look at crontab content

Any network equipment in the middle that would be rebooted, disconnected or not available for whatever reason ?

[Daniel Marques]

When it appens, zentyal GUI is very slow or freeze.
I have to make a hard switch off.
Turn it on solve the problem but i have to kill zarafat and start dovecot service manually to get my Email running.
I am not sure both problems are linked.

I am not at my office now and will come back thursday and check crontab and zentyal log.
A problem with an equipment on the network steel possible, but how to find it, we have more than 200 clients on our network.

 

[christian]

wow, that's quite confusing 
As fat as I understand, Zarafa and Dovecot do not got well together.
I also don't understand why you would manually stop/start processes when you can (should) do this using Zentyal GUI. Well, I understand GUI is slow but priority here is definitely to look for abnormal behaviour rather than rebooting. You're not running Windows platform 

Next time you face similar problem (tomorrow morning  ), have a look at system figures like CPU usage, I/O wait, swap...
I' afraid that between 7:00 and 9:00, you have bunch of users authenticating on their workstation and you have perhaps configured these clients with roaming profile.

[Daniel Marques]

Hi Christian,
Maybe I am not clear.  try to resume.
Every morning it was as the network stop running (no access to internet, clients can't connect each other, no access to mail server) clients loose or can't get  IP adresse.
Zentyal GUI is very slow ou freeze.

After the shutdown/start, clients retrieve an IP adresse can access internet or connect to other clients. All works exept Email server  whereas on the GUI message was indicated as running. Restarting the message service on GUI doesn't fixe the problem.

It was like zarafat prevent dovecot running.
The only solution i fund is killing zarafat and restart dovecot.

While we don't reboot zentyal we don't have mail problem.
But as zarafat is stopped, can it cause the troubles we get in the morning?

Hope to be more clear.
 

[Lonniebiz]

This may not be your problem, but:

Be sure you have no unknown dhcp serving devices on your network. For example, if such a device came online each morning, it might answer dhcp requests faster than your Zentyal server assigning incorrect networking configurations.

Next time this happens, be sure to do an

Code: [Select]

ipconfig /all on a windows workstation and see if it is getting assigned incorrect networking configuration; this might indicate there is some other dhcp serving device fighting to serve your network.

[Daniel Marques]

Thanks Lonniebiz,
I'll check it next time.

Today exploring monitoring curves i fund that zentyal is very busy during the night
This strange high activity start at 2am and stop 8.30am when i reboot the system and it occured every night.

maybe it's a clue
Our server was blacklisted as a spamserver. Can zentyal send massive spam every night? How to check it?

[christian]

I don't know what's the root cause but based on graphics you are showing, I would look closely at memory usage.
High CPU and system locked are, to me, only a side effect of high system load due to I/O wait.
I doubt that sending lot of messages can consume this amount of memory but this is obviously worth to check twice.

Look first at top command instead of rebooting.

[Lonniebiz]

Regarding being blocked as a spammer, from the internet's perspective do the workstations on your LAN share the same public IP as your mail server? If you have more than one public IP, put your mail server on a different public IP than your workstations use. If one of the workstations get infected by a virus, it may try to send out email directly to the internet. Since your email server is sharing the same public IP as your workstations, to the other email servers on the internet it appears as though your email server is doing the spam, and your public IP gets black listed. If you have another public IP you can use, I suggest moving your email server to talk on a fresh IP.

Assuming you are also using Zentyal as your gateway, to determine which workstation (if any) are infected, try this command:

Code: [Select]

sudo apt-get install jnettop ; sudo jnettop -i eth1 ### assuming eth1 is you inside interface
Try to see which workstations seem to be sending a lot of traffic for no reason.

If Zentyal is not your gateway, you'll have to figure out how to do this type of monitoring somehow from what ever is your gateway.

[christian]

Regarding being blocked as a spammer, from the internet's perspective do the workstations on your LAN share the same public IP as your mail server? .../...

Although it does exist, I believe quite few Zentyal deployments are designed with Zentyal not being the network gateway. This is also very unlikely that workstation on the LAN is sending mail directly. Reason is that it would require to write virus embedding all the "sendmail like" capability in order to act as an MTA. Feasible but quite heavy.
This said, as Lonniebiz explains, what matters is to find who/what is generating such mail traffic that is tagged as spam. If you don't cure it, do not try to be removed from spam list because you will be stamped as spammer again very soon.

In addition to network analysis (that is quite low level  ), you could (should) also look at your log files: standard Zentyal deployment assumes that Zentyal is your local MTA and therefore relays all outgoing mail traffic.
BTW, did you ensure that your Zentyal configuration is not set to act as open mail relay?

Workaround with secondary IP will work.
Another, perhaps easier, option is to use smart relay, e.g. MTA from your ISP. 

[toolman1967]

How you tried using a open relay test web site that checks your mail server to verify that it is closed to relay of mail?  Do you any Relay policy for network object set up, if so make that it is set to only your local LAN network and that ISP side.  Is someone using a Bit-torrent program on one of the workstation and they have it set to start downloading during the night.  The WOL functions are enabled by default on the network interface install ETHTOOL from terminal session and  run "ethtool eth0" if see Wake-on: g  than run this command "ethtool -s eth0 wol d"   (d-disable. g-wake on magic packet) to see if that helps.

[Daniel Marques]

Thanks for your comments

Zentyal is our gateway and MTA.
we only have one IP adresse

Today is a national holliday and office is closed. exepte for me.
At 8Am when i arrived zentyal was as usualy bloqued/freeze.
after 2 hours trying to connect with ssh or with abrowser, i got a ssh connection but very slow.
type top commande see attachments
I notice duplicity is taking a lot of memorry, dont know what it is.
GUI cames alive, i can move the mouse, but can't access zentyal interface
ssh connection steel very slow
 
AS i am alone, i disconnect my zentyal LAN and connect directly my macbook.
like that, I am the only one client of Zentyal.

And unfortunatly Zentyal steel high loaded.
ssh connection is difficult (i have to fait a long time before being connected)
No access by browser connection
Can't lauch zentyal interface localy

One thing is running, i have access to internet

Now i am trying to install jnettop

 

[Daniel Marques]

And this is the jnettop printscreen

192.168.0.11 is our ftp server  (i was obliged to reconnect it, for our costumers)
192.168.0.249 is my notebook
192.168.0.251 is zentyal

Nothing very heavy

[Daniel Marques]

Around 03PM, zentyal high activity reduced to the normal.
I can connect to it in ssh and with a browser normally.
the local GUI also work normally.
Top command no more shows duplicity activity.
Is duplicity the backup service as i understand?
zentyal start its backup at 2am, when the high activity start.
but why duplicity stop working at 3pm whereas zentyal log said ZEntyal incremental backup succeedded at 06:40am

and in any case this can't explain why my server was blacklisted as spam server?

[Lonniebiz]

From what I've noticed, Zentyal automates backups using duplicity commands (under the web interface).

Another thing to check, regarding your email blacklist issue, see if you are some how allowing your Zentyal server to act as an smtp smart-host for other machines on  your network.

I don't know how to check this in Zentyal yet, but in IIS SMTP, it can be set as a smart host in a way where authentication is not required for a workstation (or device) to pass emails through this IIS SMTP. In this case, if an infected workstation locates this open smtp server, it can send spam through it ultimately getting your ip blacklisted.

Again, I never give my email server the same public IP as the workstations use through NAT. That way, if some virus installs a rough email server onto a workstation, the spam is sent from a different IP than the official email server. If that IP gets blacklisted it has no effect on the email server's public IP.

Additionally, in the firewall, I block port 25 outbound for all machines except the email server.

Even after all this, you still need to locate the source of the spam that has gotten your IP blacklisted, so that you may prevent that source from using resources unnecessarily.