Error Restoring Database

[homechoiceTS]

[2013-03-06 05:46:16] Version [1.6.3.18]Archive error: Error restoring database: Operation not permitted

[Belinda Frick]

Please send a screenshot of the disk space - see forum post:

How do I check disk space available on AMETHYST server?

[Belinda Frick]

See available diskspace for 20130306:

HomeChoice-20130306-available_disk_space.jpg

[Belinda Frick]

All your exclusive drives are full. The sweeper stopped and cannot load any recordings into the database.

The recordings sent to AMETHYST will be kept in the database drive untill 1 GIG space is left. Currently 8 GIG is left. Your database size is 8 GIG and therefore the auto archive at 21h00 will not take place. Once space has been cleared on the Exclusive drive and the sweeper is running again it will clear space. Archiving will then take place again.

Currently the 'days to keep a live' is set to 670 days. Please confirm if you may change this to less days. We can then start a manual purge to clear space on the exclusive drive. Note: only archive recordings will be purged.

Second option is additional storage - if this is an option will depend on the current server specification.

Please let me know which option you decide on.

[Belinda Frick]

Once the auto purge has been configured, you need to use the AMTListen to search for recordings older than the 'days to keep alive' setting - refer to forum post:

Listening to an archived & purged recording in Amethyst 1.5+

[Belinda Frick]

Nuyanda, we still need to configure the auto purge with new 'days to keep alive' value and start a manual purge. This will clear space on the exclusive drives and allow recordings to be processed. Recordings needs to be swept to clear space so that archiving can take place again.

Currently no recordings made since the sweeper stopped will be available in Supervisor to playback or do AQMs on.

We need to do the manual purge to clear space, once completed and space cleared, start the sweeper to load recordings.

Once recordings can be swept from the Database drive to the Exclusive drive, it will clear space on the database drive. The database is about 8 GIG. It must be copied to do the backup as part of the archive. We need 16 GIG on the database drive before the archiving can take place again.

Only archived recordings can be purged. We need to get the archiving up to date. Once the auto archive runs again, the auto purge can take place each night at 00h00 to maintain space on the exclusive drives.

Please contact me so that I can assist with the configuration of the 'days to keep alive' and give instructions to start the manual purge.

I am not sure how long the manual purge will run. Only thereafter can we start the sweeper. If the back log of recordings is swept before 21h00 tonight, then the database drive will have enough space to do the auto archive.

[Belinda Frick]

Luyanda, I would like to help you start the manual purge as you could experience space issues.

Please do a Check Available Disk Space to see what the current database drive space available is. AMETHYST will stop receiving recordings via FTP from the USBClient PCs if the space is low on the AMETHYST server(we check for 1 GIG).

It would be best if we start the manual purge today.

We need the Exclusive drive space that the purge will free to start sweeping recordings from the database drive.

The sooner we start the process the better, as there is a backlog that must catch up.

[Belinda Frick]

The ‘days to keep alive’ has been configured from 670 to 365 day. Manual auto purge has been started. This means call archived recordings older thant 365 days must be purged from the 3 Exclusive drives. This comes to about half the space on these drives to be cleared. We are not sure how long this process will take – it is purging recordings for a year.

Once the manual purge is completed the sweeper must be started to process recordings.

To see if the auto purge has completed, please log into AMETHYST webadmin and go to View logs, select todays date and then ‘admin’ for the log. If you do notsee the complete entry, it completed before 24h00, check the previous days log. Now go to the Home page and click on the Available Disk Space link. You will see the newly configure days to keep alive value at the bottom and for exclusive drive the space available should not be 0 %.

If the purge was successful and the exclusive drives shows space available, go to the Home page and click on the ‘start sweeper’. If not available go to the ‘View Call History’ link and check recordings processed in the last couple of days to see if it is already started. You can also use the View Call History link to check every now and again what the progress of recordings being loaded are.

Note that if the sweeper was not running at 21h00 20130307 when the auto archive takes place, it still would not have enough space to run the archive and you will receive the same critical e-mail.

Please post a screenshot of the current Available Disk Space.

[Belinda Frick]

The purge is still in progress. It is busy purging all recordings older than 365 days. The speed of this process will depend on the speed of the system. For example if the harddrives are slow it will take longer to purge the recordings from it.

The space of the database drive went down to 600 Meg. I started the sweeper to start moving recordings to the space that was already cleared on the Exclusive drives.

Your database is now growing again. The current size is 9 GIG. This means the database drive must have about 19 GIG for the archive to be successful.

While the purge is clearing space on the exclusive drive and the sweeper is loading the backlog of recordings the AMETHYST system will be slow. We cannot speed up this process.

PLEASE NOTE: If the Supervisor search for recordings it might look like some recordings are missing. This is because not all the backlog of recordings has been loaded.

[Belinda Frick]

To explain further:

Before the auto purge was started on 20130307, this was the drive space allocation as per 'Available Disk Space' option(screen shot you sent us):

Code: Select all

/dev/sda1              31G   23G    7.4G  76% /
/dev/sda3             428G  427G  547M  100%  /mnt/drive2
/dev/sdc1             276G  275G   82M  100%  /mnt/drive3
/dev/sdb1             1.8T  1.8T  1.6M  100% /mnt/drive4

NOTE:

1. 100% indicates no space available. This was the case for all 3 exclusive drives. Therefore no recordings could be swept in to the AMETHYST database for access via AMETHSYT Supervisor.

The drive allocation on 20130308 @ 09H20 was:

Code: Select all

/dev/sda1              31G   30G  635M  98% /
/dev/sda3             428G  201G  223G  48% /mnt/drive2
/dev/sdc1             276G  247G   29G  90% /mnt/drive3
/dev/sdb1             1.8T  1.8T  1.6M 100% /mnt/drive4

NOTE:

1. Purge was only done on drive2 and drive3. Intotal 250GIG ( 222G + 28G) was clear from 20130307 15h00 until 20130308 09h20, which comes to about 19 hours for 250 GIG. Total exclusive drive spaces comes to 2.5 Terabytes (1.8 terabyte + 278 GIG + 428 GIG). Auto purge was configured from 670 days to 365, which means recordings for half the number of days must be purged - this is up to 1.25 terabytes. At the current rate it took 19 hours for 250 GIG. 1 terabyte must still be purged, which means 4 times 250 GIG. It could take up to 4 days.
   
   
2. The database drive space also started filling up from 7.4 GIG to 635 meg. This is where the recordings are stored before swept to the exclusive drive. If no space on the database drive, the recordings will stay on the agent PC. When the agent PC is full, no recordings can take place for that agent. For this reason we started the sweeper to start uploading recordings to the database and sweep it to the exclusive drive.
   
   
3. Currently recording being swept cleared space on the database drive to 882 GiG. The drive allocation on 20130308 @ 12H55 was:
   
   Filesystem Size Used Avail Use% Mounted on
   

   Code: Select all

   /dev/sda1              31G   29G  882M  98% /
/dev/sda3             428G  155G  269G  37% /mnt/drive2
/dev/sdc1             276G  247G   29G  90% /mnt/drive3
/dev/sdb1             1.8T  1.8T  1.6M 100% /mnt/drive4

   
   NOTE: only space on drive2 has been freed since last check (about 46 GIG)
   
   The database size is 9 GIG. For archiving to take place we need to make a backup of the database and therefore need 19-20 GIG free space on the database drive. Once all the back log of recordings is swept this should be available. Only then will archiving be able to take place again. You will receive a critical e-mail each night at 21h00 until the space is sorted out.

Conclude:

The auto purge is still busy. We cannot speed up the process. This will influence the speed of the system from a user point of view as well as the information the Supervisors are able to retrieve.