Staging volume full after takeover and giveback for ONTAP 9

Applies to

• Clustered DATA ONTAP 9.0P3
• Clustered DATA ONTAP 9.1P2
• Clustered DATA ONTAP 9.1P6
• Clustered DATA ONTAP 9.3
• Clustered DATA ONTAP 9.3P3
• Clustered DATA ONTAP 9.3P4
• Clustered DATA ONTAP 9.3P5

Issue

Some of the symptoms seen are as follows:

• Recent ONTAP Upgrade was done from ONTAP 9.x to 9.y.
• No audit logs reported after any takeover/giveback operation which was done during the ONTAP upgrade.
• Some of the staging volumes have reached full capacity.

EMS logs :
[?] Wed Aug 29 07:34:57 BST [cluster-2: wafl_exempt04: wafl.vol.full:notice]: Insufficient space on volumeMDV_aud_fdb78598bd5945ffa6f7bd1197a9f975@vserver:7a9e6a27-aa87-11e6-96cc-00a098a91817 to perform operation. 4.00KB was requested but only 1.00KB was available.
[?] Wed Aug 29 07:34:57 BST [cluster-2: AuditWorkerThread02: adt.stgvol.nospace:EMERGENCY]:Audit subsystem internal error: Staging volume MDV_aud_fdb78598bd5945ffa6f7bd1197a9f975 is full.
[?] Wed Aug 29 07:59:30 BST [cluster-2: AuditWorkerThread00: ems.engine.suppressed:debug]: Event 'adt.stgvol.nospace' suppressed 90 times in last 1397seconds.
[?] Wed Aug 29 07:59:30 BST [cluster-2: AuditWorkerThread00: adt.stgvol.nospace:EMERGENCY]: Audit subsysteminternal error: Staging volume MDV_aud_fdb78598bd5945ffa6f7bd1197a9f975 is full.

cluster ::*> df MDV*
Filesystem              kbytes       used      avail capacity  Mounted on
/vol/MDV_aud_4a9d8065eac9454bbe042ffddd0df645/    1992296        532    1991764       0%  /vol/MDV_aud_4a9d8065eac9454bbe042ffddd0df645/
/vol/MDV_aud_62a9aebc8f3d4fe2990e39bb34c66999/    1992296        384    1991912       0%  /vol/MDV_aud_62a9aebc8f3d4fe2990e39bb34c66999/
/vol/MDV_aud_fdb78598bd5945ffa6f7bd1197a9f975/    1992296    1992296          0     100%  /vol/MDV_aud_fdb78598bd5945ffa6f7bd1197a9f975/ <<<<<< Full
/vol/MDV_aud_d63c9c461ab14609928999846af40f90/    1992296        192    1992104       0%  /vol/MDV_aud_d63c9c461ab14609928999846af40f90/
/vol/MDV_aud_a38d6a453935479d8e0a5a5413f986c3/    1992296        204    1992092       0%  /vol/MDV_aud_a38d6a453935479d8e0a5a5413f986c3/

Disabling Audit fails:
cluster::*> vserver audit disable -vserver svm01
Error: command failed: Failed to disable auditing for Vserver "svm01". Reason: Audit consolidation job not present for the Vserver and one of the possible reasons might be snapmirror break operation in Vserver DR setup. Retry after some time.

CIFS clients will face an outage if Strict Guarantee of Auditing is enabled:
cluster::*> vserver audit show -vserver  svm01 -fields audit-guarantee
vserver   audit-guarantee
--------- ---------------
svm01   true