Author Topic: OpenChange causes Samba crash (and then no more usable)  (Read 13174 times)

jecheve

  • Zen Apprentice
  • *
  • Posts: 6
  • Karma: +1/-0
    • View Profile
OpenChange causes Samba crash (and then no more usable)
« on: March 25, 2014, 04:13:58 pm »
Hi everybody,

I will try to explain my best the very hard problems I have found installing and configuring Openchange in Zentyal 3.4 (those and a few more in Zentyal 3.3)

I start from a fresh and clean install, choosing "OpenChange Server". Everything goes OK. Domain DC, users, shares, etc... Creating a user, creating an e-mail account, and enable OpenChange mailbox is OK. In fact, I create a profile in Outllok  2007 and everything goes well. Although automatic account configuration does not work (it discovers the server and my e-mail account as I am logged into the domain, but then fails), but manual configuration runs smoothly. I open Outlook and first use goes OK (although a curiosity: after first login folder tree root is called "Mailbox - Full user name"; second and then it is called: "OpenChange: username".

Well, I start trying. I can send and receive mails to/from external accounts and to/from other OpenChange accounts.

The problems begin when I Import something from a .PST (of course I want to keep my old e-mails). Samba crashes! Im fact I get a message in Dashboard about a crash of usr_sbin_samba. Samba and Openchange need to be restarted. But after restarted you can access any other OpenChange mailbox, but not the one that caused the crash. If you use other mailbox, you can connect, send, received and stay connected. If you import something or try to access a mailbox that previously crashed the system, it crashes again, and you need to restart Samba and OpenChange. Sometimes, the whole system.

Apart than the problem itself, i have tried to clean or remove the OpenChange mailboxes, but no success. If I disable an OpenChange account, delete the e-mail account and then create a different account for the user and enable the Openchange Mailbox, it takes the old "corrupted" mailbox. I have tried to unconfigure OpenChange, but I am said "server cannot be unregistered, something like a primary update service relies on it". I have disabled the OpenChange module, and the deleted. Then installed it and enabled. And provisioning it again (with the previous organizations; the system knos about it); and I can access the mailboxes but when I do anything, send a single message, etc... another crash!

Is there any way of cleaning, removing or deprovisioning OpenChange? Or at least removing (not only disabling) a Mailbox?

I know you are going to ask me for reproducing the problem (it has happened to me a lot of times) and send logs and crash files, but I have no way of reproducing the initial state unless reinstalling the whole system, but I have it already working as a primary AD controller, and domain and file sharing services run smoothly... (unless  they crash when importing something into OpenChange).

Please, help (I am going mad with this issue). For the last two weeks I have re-installed Zentyal, created a domain, joined 10 computers, created users, take out computers our of the domain, begin again from scratch... And again, and again... First with 3.3, then 3.4

I thought it has relation with a bug opened about importing s/mime messages, but no. It fails as soon as you import anything, and remains unusable.

Thank you everybody in advance. Regards,

Javier

jecheve

  • Zen Apprentice
  • *
  • Posts: 6
  • Karma: +1/-0
    • View Profile
CRITICAL: Samba crashes and OpenChange no more usable
« Reply #1 on: March 26, 2014, 07:39:51 pm »
I have suffered this issue so many times in the last two weeks, and it is so easily reproducible, that I am going to tell the story step by step and offer all my logs, crash files, even access to my system to Zentyal staff/developers if interested. I think it is a CRITICAL issue, because, in fact, makes OpenChange deployment not an viable option.

Summary: some import of messages (almost any) and/or folder operations (create/delete) in OpenChange mailbox causes Samba to crash. Outlook connection aborts, and it is impossible to reconnect again. Samba and or OpenChange restart required. Moreover, then if you get connected it aborts and Samba crashes again and again, leaving OpenChange mailbox or perhaps database schema unusable.

Step by step procedure:

* We create a bootable live USB key from Zentyal 3.4 ISO i386, 32 bits. (It also happens with 64 bits version)
* Boot our server from this key, we choose Español (Spanish) language ans select Install (expert mode) (also happens with other install methods. tested)
* Country: España. Keyboard config detection: No. Keyboard: Español. Keyboard distribution: Español
* Hostname: servidor
* username: administrador (password and confirm password)
* Accept "Europe/Madrid" as TZ
* Warning about installation media mounted on /dev/sdb1 - Accept
* Partitioning: Chose "Guided - use whole disk and configure LVM"
* Chose our disk: SCSI8 (0,0,0) (sda) - 80.0 GB SEAGATE ST380011A
* Save changes and configure LVs? Yes. Delete previous LVs? Yes. Use maximum space (79.8 GB). Again, warning about installation media on /dev/sdb1. Accept. Write new partition table for sda? Yes.
* Install without GUI: yes (no GUI)
* No proxy
* Install GRUB in MBR? Yes
* System clock in UTC? Yes
* Installation completed. Extract installation media, etc. Continue
* System reboots. Installing core packages
* Zentyal interface at https://192,168.1.36 (given by DHCP server in my ADSL router)
* Access from browser in my laptop, select OpenChange server and install. Accept selected packages to be installed.
* Configuration: eth0 as internal. Static IP: 192.168.1.10   Mask: 255.255.255.0   Gateway: 192.168.1.1 (my ADSL router) DNS Server 1: 192.168.1.10 (Zentyal server itself).
* Select Standalone AD server. Domain: utilisaction.local
* Virtual mail domain: utilisaction.local
* Server registered in Zentyal Remote as: utiliserver.zentyal.me
* Installation... configuration... success
* Created user "dadmin" (Domain Administrator), an included in group "Domain Administrators"
* Created user "javier" (Javier Echeverría)
* New mail virtual domain added: utilisaction.eu. Changes saved
* Openchange provisiones: Organization name: utilisaction. Changes saved.
* Domain utilisaction.eu configured as the one for outgoing e-mail. Changes saved
* Mail: Smarthost added (mail.utilisaction.eu:26) User/password autentication (javier@utilisaction.eu) LOGIN method. Not necessary but for avoiding rejections due to the lack of an SPF or having to include an SPF record in external domain DNS servers. Changes saved
* Enabled retrieving mail from external accounts. saves changed.
* For user "javier": Retrieve messages from external account javier@utilisaction.eu (mail.utilisaction.eu IMAP, port 143). OpenChange account enabled (for its email account javier@utilisaction.local, created when creating the user as per the user creation template)
* My Windows 7 laptop joins the domain (using "dadmin" user credentials).
* Restart my laptop. Login as "javier"
* Shares, domain services... all run OK.
* Open Outlook 2007.
* Configuration assistant detects name (Javier Echeverría) and email account (javier@utilisaction.local)
* Click next. Credentials, security warning about certificate. Accept
* Automatic configuration fails (shouldn't it work? I think so)
* OK, no problem. Manual for server options. Choose Exchange. User populated with name (Javier Echeverría). Server: servidor.utilisaction.local. Verify name. Success ---> underlined. Next Finish.
* Configuring Outlook for its first use... Asks for name and initials... Online help options... Outlook is configuring a local copy of mailbox. Ask "Synchronize RSS sources". Yes
* Connected to Microsoft Exchange and everything updated and Sync'd
* Folders (after this first login. Interesting, because then they change):
    Buzón - Javier Echeverría
        Correo electrónico no deseado
        Drafts
        Elementos Borrados
        Fuentes RSS
        INBOX
       Outbox
       Sent
       Spam
       Carpetas de búsqueda
  (some English, some Spanish)
* Close Outlook / Open Outlook. Mailbox root folder name changes to: "OpenChange - javier". Rest the same (mixed Spanish and english)
* Everything is OK. Test e-mail message sent to jecheve@movistar.es
* First appear in Ourbox, then in Sent.
* WARNING: Initially in "Sent" with correct hour (15:57) and inmediately (when folder syncs, I suppose) it changes to 14:57!
* Visit my movistar.es webmail. Correctly received from from Javier Echeverría <javier@utilisaction.eu> with correct hour (15:57)!
* Answered from movistar.es webmail
* Reply correctly retrieved and punt in OpenChange INBOX. Hour is OK.
* Anyway, if you choose "Message properties", section "Internet headers" is empty and it shouldn't. There is a bug in tracker about it...

Up to here everything runs (almost) perfectly (but autoconfiguration, time handling and Internet headers). I have repeated this dozen times, and I can send/receive any message to/from any external or local account (including attachments, etc.)

The nightmare begins when I have tried to import messages (and contacts, calendars, etc.) from an exported backup (.PST) from the previous Outlook POP3 profile into the OPenchange mailbox. This process always makes Samba crash, disconnect Outlook from Exchange, makes impossible to reconnect, requires Samba and/or Openchange restaring, then you can connect again but inmediately of after any minimal operation, Samba crashes again, you get disconnected and not only the specific mailbox but others (the whole Database schema, I suspect) remains unusable. Tried dozen times from scratch, I repeat. By the way, sometimes after several crashes all folders have appeared with its name in Spanish!

In this ocasion, for being able to see something, instead of importing a whole .pst I have decided to go slower, step by step:

* Export from another Outlook a single folder (Terra) with a single message.
* In Openchange Mailbox, I create a folder called "Old messages".
* Import only messages in the exported folder (not the whole folders tree: Personal Folders -> Inbox -> Terra) into current folder (Old messages), not directly in OpenChange:javier. OK, message imported. Although status bar says indefinitely " awating to be updated" when it is really updated and sync'ed (seen in folder properties). Although you click Send&Receive, etc message in the status bar does not change to "up to date" until you close and open Outlook again. although it is really synced. Well, a minor issue.
* Deleted the single message in "Old messages Folder". Now we import th whole structure. Goes OK. Under "Old messages" folder:
   Bandeja de Entrada
       Terra
           Message
* Then, I import a folder with several messages from another .PST into "Old messages" folder. And everything goes OK.
* I was going to import a full .pst backup into the "Old messages" or directly into the mailbox (what has also caused the carsh and the explained disaster), but previously I decide to delete all content in "Old Messages" folder and goes OK, and then the "Old messages" folder itself. And... BUMP. Samba crashes, Openchange aborts, Outlook disconnect... and the disaster. An unusable Openchange provision.

So i think, it has relation to creation/deletion of folders or elements in the root folder of the Openchange Mailbox, not exactly the import of folders or elements (what obviusly implies the first).

Cannot attach zentyal.log, samba.log or crash file (although sent through Zentyal web admin) because they are bigger than allowed.

And, of course I can grant access to the crashed system. SSH and HTTPS ports in my router are forwarded and I can change passwords for Zentyal staff/development

I think issue is critical for OpenChange (by the way also happened with 3.3 and I was awating to see if solved in 3.4)

Thanks in advance. Regards,

Javier

jkerihuel

  • Zentyal Staff
  • Zen Warrior
  • *****
  • Posts: 152
  • Karma: +18/-3
    • View Profile
Re: OpenChange causes Samba crash (and then no more usable)
« Reply #2 on: March 27, 2014, 01:05:09 pm »
Hola Javier,

Thanks for your feedback and detailed response. It is much appreciated.

One of our engineer is going to have a deeper look at look and get back to you shortly with updates, questions or eventually help inquiries to move this matter further.

Br,
Julien.
Twitter: http://twitter.com/jkerihuel
Key fingerprint = 08BA 50B1 9EFF 8E1E FB4A  24FA B2A9 D5F3 9624 1CC2

blaxter

  • Zentyal Staff
  • Zen Apprentice
  • *****
  • Posts: 10
  • Karma: +2/-0
    • View Profile
Re: OpenChange causes Samba crash (and then no more usable)
« Reply #3 on: March 27, 2014, 01:33:50 pm »
thanks for the great report Javier and sorry for the inconveniences :(.

First the easy minor thing about the change of name of the mailbox, that's normal and it happens for outlook locale different from English, actually the way to go will be to execute outlook (win+r, enter) as "outlook.exe /resetfoldernames" so you have all names on your language.

Another thing, about deleting an user and creating it again is a known bug that I wanted to fix before 3.4 release but I didn't have the time, it should be fixed "soon" (you can add you as a watcher to that ticket if your are interested)

And now the main issue, importing messages. We have a ticket related with that, maybe is that, maybe is another thing. Could you do a grep in your samba.log files? Something like this:

Code: [Select]
# zgrep Uncaught /var/log/samba/*
# zgrep "Assertion failed" /var/log/samba/*

The first is to see SOGo crashes, the second is to check if is the reported bug about importing messages.

jecheve

  • Zen Apprentice
  • *
  • Posts: 6
  • Karma: +1/-0
    • View Profile
Re: OpenChange causes Samba crash (and then no more usable)
« Reply #4 on: March 27, 2014, 08:21:55 pm »
thanks, blaxter for the tip on resetting Outlook folders. I will try it later.

About the important issue, there are no entries as the ones you suggest in samba logs. I think it is not really the same issue as in the bug about importing messages with s/mime signatures. In fact I began suffering this crashes whe importing messages, but in the example told yesterday, import went OK, deletion of imported messages went OK, and crash occured when deleting the folder into which messages had been imported. But I have searched the logs in another server, in which first crash happened when importing a quite big amount of messages (a .PST with a full backup of a real Outlook profile) and there are no messages as those you suggest.

Another question making me suspicious is the fact that, although logged into the domain (and so Outlook detects name and mailbox) automatic configuration and login fails. And when server manually entered (and name correctly verified) it asks for credentials. And it shouldn't if Kerberos were working properly. What do you think?

Thank you in advance,

Javier

jecheve

  • Zen Apprentice
  • *
  • Posts: 6
  • Karma: +1/-0
    • View Profile
Re: OpenChange causes Samba crash (and then no more usable)
« Reply #5 on: March 28, 2014, 02:23:17 am »
Hi, Blaxter (and all)

I have started again from a fresh install, etc... Joined my laptop to the domain, login with my brand new user, and opened Outlook. Although automatic configuration detection and login didn't work, adding the server manually worked. As I started Outlook with /resetfoldernames all the folders have their correct names in Spanish (thanks!)

Well, connected to Exchange and everything OK. I have created a new folder "Imported", and from a backup.pst file I have imported only the three messages in one folder to that "Imported" folder. Apparently, everything OK. But Outlook has remained indefinitely awaiting to update folders... I have tried Send/Receive but it has got stuck. After several minutes I have had to force Outlook to close. Then, when opening again (several times) it has been impossible to connect to OpenChange. I have restarted my laptop, and impossible to connect. (but no samba crash). Finally I have restarted Zentyal server and I have been able to connect to Exchange, but it has almost inmediately disconnected. And... voila a crash of samba announced in dashboard. Impossible to connect. Now we are in a state, very well known to me, where if I restart samba I can connect, but it inmediately crash after receiving a connection from Outlook, and again, and again. And caused after importing 3 messages in a new folder!

Of course, I can provide you any log or crash file/itself. Also, if you want I cant grant to you access to this system, for you to analyze or test whatever.

Br,

Javier

blaxter

  • Zentyal Staff
  • Zen Apprentice
  • *****
  • Posts: 10
  • Karma: +2/-0
    • View Profile
Re: OpenChange causes Samba crash (and then no more usable)
« Reply #6 on: March 28, 2014, 12:05:59 pm »
About the autonconfiguraton, right now is not working properly (it should, but is normal the problem you are having).

About the crash, it seems like we have a new bug related with importing messages, can you upload the samba.log file? You are now in a state where samba crashes whenever you open outlook, so if samba.log now is too big, just do a echo "" > /var/log/samba.log and then open outlook.

jecheve

  • Zen Apprentice
  • *
  • Posts: 6
  • Karma: +1/-0
    • View Profile
Re: OpenChange causes Samba crash (and then no more usable)
« Reply #7 on: March 28, 2014, 12:14:54 pm »
Hi, blaxter

Actually samba.log is about 2.6 Mb. I think it is interesting not to delete it, because it has the first problems and crash after the fresh install and importing a few messages. If we delete it, we will only have the crashes caused by the bad state in which system remains after the first crash.

I cannot upload it, but I can send it to you wherever. Also, perhaps crash file itself is of interest.

if you want, I can grant to you access to the system (utiliserver.zentyal.me). HTTPS and SSH are open and I can change and give you password.

regards,

Javier

blaxter

  • Zentyal Staff
  • Zen Apprentice
  • *****
  • Posts: 10
  • Karma: +2/-0
    • View Profile
Re: OpenChange causes Samba crash (and then no more usable)
« Reply #8 on: March 31, 2014, 09:55:01 am »
hi again,

you could gzip the file. Or you can join irc (freenode #zentyal, I'm blaxter there, normally I'm ~8-19h utc) and we can talk there. The problem as you describe seems to be easily reproducible, so I just want the log so I can open a bug ticket on our tracker.zentyal.org

jecheve

  • Zen Apprentice
  • *
  • Posts: 6
  • Karma: +1/-0
    • View Profile
Re: OpenChange causes Samba crash (and then no more usable)
« Reply #9 on: March 31, 2014, 10:37:37 am »
Hi, blaxter

I will reproduce the situation later today and proceed as you suggest. I have been experimenting during the weekend and the system is quite a mesh.


blaxter

  • Zentyal Staff
  • Zen Apprentice
  • *****
  • Posts: 10
  • Karma: +2/-0
    • View Profile
Re: OpenChange causes Samba crash (and then no more usable)
« Reply #10 on: April 01, 2014, 03:25:36 pm »
After seeing the logs and try this for myself I've created a bug that I think is the root cause of the issues you had importing messages: https://tracker.zentyal.org/issues/516 you can add yourself as a watcher if you want to track it. I don't know when will be fixed though :(.

b.deivid

  • Zen Apprentice
  • *
  • Posts: 25
  • Karma: +3/-0
    • View Profile
Re: OpenChange causes Samba crash (and then no more usable)
« Reply #11 on: April 16, 2014, 08:22:30 pm »
I'm experiencing the same behaviour !  I have tried to import the PST file using Outlook into OPENCHANGE.
And, what happened ??  Samba CRASHED, and CRASHED, and CRASHED, and CRASHED ..... Had to reinstall zentyal.

Bug over Bug, over BUG, over BUG.
« Last Edit: April 16, 2014, 08:31:02 pm by b.deivid »

Ludwig

  • Zen Apprentice
  • *
  • Posts: 3
  • Karma: +0/-0
    • View Profile
Re: OpenChange causes Samba crash (and then no more usable)
« Reply #12 on: April 23, 2014, 12:10:08 am »
Hi all,

It seems, that OpenChange does not like Outlook Exchange Cache Mode (as well as local .PST Folders) at all!
I've been struggeling for days with this issue now, but found a very easy "workaround" this evening:

Just turn off Outlook Exchange Cache mode.
This prevents Outlook from creating al locat .OST file (for offline mails) and as an effect also avoids any MAPIStore********* - operations, which seem to be the root cause of Samba-Crashes in conjunction with OpenChange.

Sure, that hint/workaround is not operable in productive environments, but it might help to narrow down the search... (I've seen at least 10 somehow related newly reported bugs at https://tracker.zentyal.org/projects/zentyal/issues?set_filter=1&tracker_id=1, which might be "solved" by turning off cache mode.)

Cheers and happy bug tracking,
Ludwig
« Last Edit: April 23, 2014, 12:23:46 am by Ludwig »

psifactory

  • Zen Apprentice
  • *
  • Posts: 10
  • Karma: +0/-0
    • View Profile
Re: OpenChange causes Samba crash (and then no more usable)
« Reply #13 on: April 24, 2014, 03:12:48 pm »
is there any workaround to fix this problem? Without deactivating the Exchange Cache Mode?

Ludwig

  • Zen Apprentice
  • *
  • Posts: 3
  • Karma: +0/-0
    • View Profile
Re: OpenChange causes Samba crash (and then no more usable)
« Reply #14 on: April 25, 2014, 12:23:51 am »
Hmmm,

got some (but much less) crashes since turning off - so "Workaround disable cache mode" obviously solves some crashes, but unfortunately not all  :(
Therefore the information regarding use of cache mode is intentially aimed to development in order to narrow bug tracking and filter open issues/requests.

Cheers,
Ludwig