Staging volume full after takeover and giveback for ONTAP 9
Applies to
- Clustered DATA ONTAP 9.0P3
- Clustered DATA ONTAP 9.1P2
- Clustered DATA ONTAP 9.1P6
- Clustered DATA ONTAP 9.3
- Clustered DATA ONTAP 9.3P3
- Clustered DATA ONTAP 9.3P4
- Clustered DATA ONTAP 9.3P5
Issue
Some of the symptoms seen are as follows:
- Recent ONTAP Upgrade was done from ONTAP 9.x to 9.y.
- No audit logs reported after any takeover/giveback operation which was done during the ONTAP upgrade.
- Some of the staging volumes have reached full capacity.
EMS logs :
[?] Wed Aug 29 07:34:57 BST [cluster-2: wafl_exempt04: wafl.vol.full:notice]: Insufficient space on volumeMDV_aud_fdb78598bd5945ffa6f7bd1197a9f975@vserver:7a9e6a27-aa87-11e6-96cc-00a098a91817 to perform operation. 4.00KB was requested but only 1.00KB was available.
[?] Wed Aug 29 07:34:57 BST [cluster-2: AuditWorkerThread02: adt.stgvol.nospace:EMERGENCY]:Audit subsystem internal error: Staging volume MDV_aud_fdb78598bd5945ffa6f7bd1197a9f975 is full.
[?] Wed Aug 29 07:59:30 BST [cluster-2: AuditWorkerThread00: ems.engine.suppressed:debug]: Event 'adt.stgvol.nospace' suppressed 90 times in last 1397seconds.
[?] Wed Aug 29 07:59:30 BST [cluster-2: AuditWorkerThread00: adt.stgvol.nospace:EMERGENCY]: Audit subsysteminternal error: Staging volume MDV_aud_fdb78598bd5945ffa6f7bd1197a9f975 is full.
cluster ::*> df MDV*
Filesystem kbytes used avail capacity Mounted on
/vol/MDV_aud_4a9d8065eac9454bbe042ffddd0df645/ 1992296 532 1991764 0% /vol/MDV_aud_4a9d8065eac9454bbe042ffddd0df645/
/vol/MDV_aud_62a9aebc8f3d4fe2990e39bb34c66999/ 1992296 384 1991912 0% /vol/MDV_aud_62a9aebc8f3d4fe2990e39bb34c66999/
/vol/MDV_aud_fdb78598bd5945ffa6f7bd1197a9f975/ 1992296 1992296 0 100% /vol/MDV_aud_fdb78598bd5945ffa6f7bd1197a9f975/ <<<<<< Full
/vol/MDV_aud_d63c9c461ab14609928999846af40f90/ 1992296 192 1992104 0% /vol/MDV_aud_d63c9c461ab14609928999846af40f90/
/vol/MDV_aud_a38d6a453935479d8e0a5a5413f986c3/ 1992296 204 1992092 0% /vol/MDV_aud_a38d6a453935479d8e0a5a5413f986c3/
Disabling Audit fails:
cluster::*> vserver audit disable -vserver svm01
Error: command failed: Failed to disable auditing for Vserver "svm01". Reason: Audit consolidation job not present for the Vserver and one of the possible reasons might be snapmirror break operation in Vserver DR setup. Retry after some time.
CIFS clients will face an outage if Strict Guarantee of Auditing is enabled:
cluster::*> vserver audit show -vserver svm01 -fields audit-guarantee
vserver audit-guarantee
--------- ---------------
svm01 true