release.notes

[v7r0-pre24]

FIX: removed __init__ files from test directories to avoid importing

NEW: Change the schema of the Resources description in CS, use new Resources helper to 
     interpret this information
NEW: Base RSS resources representation on the new CS Resources schema     
NEW: RSS Pilots Efficiency Policy
NEW: Added Licence statement available in all the commands through --license switch

*Core
FIX: multiple fixes to remove pylint complaints
CHANGE: Moved log coloring to utilities so it can be used consistently across DIRAC
FIX: Workflow package - fix wild wild card imports resulting in circular imports
NEW: MySQL - assign one MySQL connection per thread
FIX: LocalConfiguration - if short option is not set, do not print "-" in help message,
     closes #1371
FIX: dirac-install - update creation of bashrc/cshrc to allow multiplatform installations
NEW: InstallTools - check inheritance of modules to be loaded to determine their types;
     executors can now be handled by dirac-setup-site
FIX: LocalConfiguration - allow '-' character in long script options, e.g. "--long-script-option"     
CHANGE: Removed "import *" from DIRAC.__init__
NEW: dirac-install allows projects to define mirrors from where to download other projects 
     (DIRAC mirrors for instance)
FIX: Removed deprecation warning when importing MySQLdb package
FIX: BaseClient - take into account DISET decorator     
CHANGE: MySQL - added okIfTableExists flag to the _createTables() method, in case of OK 
        returns a list of created tables

*Accounting
FIX: AccountingDB - align properly days with MySQL bucketing. Closes #1219
FIX: DataStoreClient - make a deep copy of the records' values
NEW: Accounting selectors will allow registered users to choose from all possible values 
     except from users and groups
FIX: AccountingDB - randomize the order type insertion in order to avoid type starvation
NEW: AccountingDB - add a monitoring record for each type in IN tables     

*Framework
FIX: ProxyDB - prevent duplicate key errors on writing VOMSProxies to DB. Closes #1228
BUGFIX: NotificationDB - missing "," in SQL, closes #1373
FIX: dirac-proxy-get-uploaded-info.py - typo Value -> Message 

*Configuration
NEW: Resources helper class to work with the new /Resources structure according to RFC #5
NEW: dirac-configuration-convert-resources-schema - command to convert old /Resources schema
     to the new one

*Interfaces
CHANGE: Job.py - setPlatform renamed to setSubmitPools

*DMS
CHANGE: MigrationMonitoringAgent - removed obsoleted
NEW: FileCatalogClient - getFileUserMetadata to get both file and directory metadata for a given file
CHANGE: FileCatalogClient - list meta data sets
FIX: FileCatalog - extra check while new user registration for already existing entry
FIX: FileManagerBase - clean the FC_FileAncestors table when removing a file
BUGFIX: ReplicaManager - pass catalog argument in putAndRegister to registerFile call, closes #1369
FIX: SRM2Storage - in getCurrentStatus() use pythonCall to impose a timeout on top of
     lcg_util.lcg_stmd call
NEW: IRODSStorageElement.py - first import     
NEW: DFC - use ObjectLoader to instantiate catalog component plug-ins
NEW: DFC - createTables according to the in-class schema definitions
NEW: FileCatalogClientCLI - added -q (quite) option to the find command

*WMS
CHANGE: JobScheduling - is now extensible. Added unit test
NEW: SiteDirector - define which extensions pilot should install in the options, do not take from globals
FIX: SandboxStoreClient - change default from False to None ( to suite pylint ) 
NEW: JobAgent - send Accounting record for Jobs Rescheduled 
CHANGE: JobDB, JobLoggingDB and PilotAgentsDB - use python table description 
FIX: Fixed JobDB.JobJDLs BLOB columns ( they had defaults, but BLOB does not accept default values )
CHANGE: JobLoggingDB schema - StatusTimeOrder DOUBLE(12,3) instead of (11,3)
CHANGE: JobLoggingDB schema - a new field has been added, SeqNum (INTEGER), which is part of the 
        new PK (JobID, SeqNum). SeqNum is generated by a trigger at every insert and behave as a counter
        within a given JobID
NEW: new Splitters framework          

*Transformation
NEW: TaskManager - if a site is specified in the job definition, it is now taken into account 
     when submitting the task
NEW: Speeding up the getTransformationSummary call, using an UpdateTransformationCounters agent     
FIX: Multiple fixes to please pylint
FIX: TransformationDB - removed bad index on TaskID
NEW: TransformationDB - added methods to DELETE rows from TransformationFIleTasks table,
     minor changes in operation order to be ready for FK.
CHANGE: field definition for TaskID in TransformationFiles table (from
        VARCHAR to INT, in order to be ready for FK definition: TaskID in
        TransformationTasks and TransformationFileTasks is INT),
        !!!!! needs update of the MySQL schema on already installed databases
CHANGE: TransformationDB - in DataFiles table removed LFN field from the Primary Key, 
        tt was already UNIQUE, Primary key is FileID only now.
*Transformation
FIX: TransformationCleaning Agent status was set to 'Deleted' instead of 'Cleaned'

*Resources
FIX: ComputingElement - properly check if the pilot proxy has VOMS before adding it to the 
     payload when updating it
NEW: StorageElement - enable Storage Element proxy configuration by protocol name
CHANGE: glexecComputingElement - updated to reflect new glexec evolution, added DIRAC_GLEXEC 
        to the environment when running under glexec     
NEW: XXXStorage - added getCurrentStatus() method        
NEW: SSHOSGComputingElement and condorgce script

*Stager
NEW: Stager API: dirac-stager-monitor-file, dirac-stager-monitor-jobs,
     dirac-stager-monitor-requests, dirac-stager-show-stats
     
*SMS
NEW: StorageManagementDB - use python table description      

[v6r10-pre15]

*Core
FIX: Several fixes on DB classes(AccountingDB, SystemLoggingDB, UserProfileDB, TransformationDB, 
     JobDB, PilotAgentsDB) after the new movement to the new MySQL implementation with a persistent 
     connection per running thread
NEW: SystemAdministratorCLI - better support for executing remote commands 
FIX: DIRAC.__init__.py - avoid re-definition of platform variable    
NEW: Graphs - added CurveGraph class to draw non-stacked lines with markers
NEW: Graphs - allow graphs with negative Y values
NEW: Graphs - allow to provide errors with the data and display them in the CurveGraph
FIX: InstallTools - fix for creation of the root@'host' user in MySQL 
FIX: dirac-install - create links to permanent directories before module installation
CHANGE: InstallTools - use printTable() utility for table printing
CHANGE: move printTable() utility to Core.Utilities.PrettyPrint
NEW: added installation configuration examples
FIX: dirac-install - fixBuildPath() operates only on files in the directory

*CS
CHANGE: getVOMSVOForGroup() uses the VOMSName option of the VO definition 
NEW: CE2CSAgent - added ARC CE information lookup

*Framework
FIX: SystemAdministratorIntegrator - use Host option to get the host address in addition to the section name, closes #1628
FIX: dirac-proxy-init - uses getVOMSVOForGroup() when adding VOMS extensions

*DMS
CHANGE: DFC - optimization and bug fixes of the bulk file addition
FIX: TransferAgent - protection against badly defined LFNs in collectFiles()
NEW: DFC - added getDirectoryReplicas() service method support similar to the LFC
CHANGE: DFC - added new option VisibleReplicaStatus which is used in replica getting commands
CHANGE: FileCatalogClientCLI client shows number of replicas in the 2nd column rather than 
        unimplemented number of links
CHANGE: DFC - optimizations for the bulk replica look-up
CHANGE: DFC updated scalability testing tool FC_Scaling_test.py        
NEW: DFC - methods returning replicas provide also SE definitions instead of PFNs to construct PFNs on the client side
NEW: DFC - added getReplicasByMetadata() interface
CHANGE: DFC - optimized getDirectoryReplicas()
CHANGE: FileCatalogClient - treat the reduced output from various service queries restoring LFNs and PFNs on the fly
NEW: DFC - LFNPFNConvention flag can be None, Weak or Strong to facilitate compatibility with LFC data 
CHANGE: FileCatalog - do not return PFNs, construct them on the client side
CHANGE: FileCatalog - simplified FC_Scaling_test.py script
NEW: FileCatalog/DatasetManager class to define and manipulate datasets corresponding to meta queries
NEW: FileCatalogHandler - new interface methods to expose DatasetManager functionality
NEW: FileCatalogClientCLI - new dataset family of commands
FIX: StorageFactory, ReplicaManager - resolve SE alias name recursively
FIX: FTSRequest, ReplicaManager, SRM2Storage - use current proxy owner as user name in accounting reports, closes #1602
BUGFIX: FileCatalogClientCLI - bug fix in do_ls, missing argument to addFile() call, closes #1658
NEW: FileCatalog - added new setMetadataBulk() interface, closes #1358
FIX: FileCatalog - initial argument check strips off leading lfn:, LFN:, /grid, closes #448
NEW: FileCatalog - added new setFileStatus() interface, closes #170, valid and visible file and replica statuses can be defined in respective options.
CHANGE: multiple new FTS system fixes
CHANGE: uniform argument checking with checkArgumentFormat() in multiple modules
CHANGE: FileCatalog - add Trash to the default replica valid statuses

*WMS
CHANGE: JobPolicy - optimization for bulk job verification
NEW: JobPolicy - added getControlledUsers() to get users which jobs can be accessed for 
     a given operation
CHANGE: JobMonitoringHandler - Avoid doing a selection of all Jobs, first count matching jobs 
        and then use "limit" to select only the required JobIDs.
NEW: JobMonitoringHandler - use JobPolicy to filter jobs in getJobSummaryWeb()
NEW: new Operations option /Services/JobMonitoring/GlobalJobsInfo ( True by default ) to 
     allow or not job info lookup by anybody, used in JobMonitoringHandler       
BUGFIX: SiteDirector - take into account the target queue Platform
BUGFIX: JobDB - bug in __insertNewJDL()    
CHANGE: dirac-admin-show-task-queues - enhanced output  
CHNAGE: JobLoggingDB.sql - use trigger to manage the new LoggingInfo structure  
        
*RSS
FIX: DowntimeCommand - take the latest Downtime that fits    
NEW: porting new Policies from integration  
NEW: RSS SpaceToken command querying endpoints/tokens that exist  
        
*Resources
NEW: added SSHOARComputingElement class        
        
*TS
CHANGE: TransformationClient(DB,Manager) - set file status for transformation as bulk operation 
CHANGE: TransformationClient - applying state machine when changing transformation status
BUGFIX: TransformationClient(Handler) - few minor fixes
NEW: TransformationDB - backported __deleteTransformationFileTask(s) methods
CHANGE: TransformationDB(Client) - fixes to reestablish the FileCatalog interface
FIX: TransformationAgent - added MissingInFC to consider for Removal transformations
FIX: TransformationDB.sql - DataFiles primary key is changed to (FileID) from (FileID,LFN) 

*RMS
NEW: Migrate to use the new Request Management by all the clients
CHANGE: RequestContainer - Retry failed transfers 10 times and avoid sub-requests to be set Done 
        when the files are failed
CHANGE: Use a unique name for storing the proxy as processes may use the same "random" name and 
        give conflicts
NEW: RequestClient(Handler) - add new method readRequest( requestname)                 

*Workflow
NEW: Porting the LHCb Workflow package to DIRAC to make the use of general purpose modules and
     simplify construction of workflows        

[v6r9p27]

*SMS
FIX: StorageManagementDB - in removeUnlinkedReplicas() second look for CacheReplicas 
     for which there is no entry in StageRequests

[v6r9p26]

*Resources
CHANGE: CREAMComputigElement - Make sure that pilots submitted to CREAM get a 
        fresh proxy during their complete lifetime
*Framework
FIX: ProxyDB - process properly any SQLi with DNs/groups with 's in the name

[v6r9p25]

*TS
CHANGE: TransformationClient - changed default timeout values for service calls
FIX: TransformationClient - fixes for processing of derived transformations 

[v6r9p24]

*TS
FIX: TransformationClient - in moveFilesToDerivedTransformation() set file status
     to Moved-<prod>

[v6r9p23]

*Core
BUGFIX: InstallTools - improper configuration prevents a fresh new installation

*WMS
BUGFIX: PilotDirector - Operations Helper non-instantiated

[v6r9p22]

*WMS
FIX: PilotDirector - allow to properly define extensions to be installed by the 
     Pilot differently to those installed at the server
FIX: Watchdog - convert pid to string in ProcessMonitor

*TS
FIX: TransformationDB - splitting files in chunks

*DMS
NEW: dirac-dms-create-removal-request command
CHANGE: update dirac-dms-xxx commands to use the new RMS client,
        strip lines when reading LFNs from a file

[v6r9p21]

*TS
FIX: Transformation(Client,DB,Manager) - restored FileCatalog compliant interface
FIX: TransformationDB - fix in __insertIntoExistingTransformationFiles()

[v6r9p20]

*Core
BUGFIX: ProxyUpload - an on the fly upload does not require a proxy to exist

*DMS
CHANGE: TransferAgent - use compareAdler() for checking checksum
FIX: FailoverTransfer - recording the sourceSE in case of failover transfer request 

*WMS
FIX: ProcessMonitor - some fixes added, printout when <1 s of consumed CPU is found

*Transformation
BUGFIX: TransformationClient - fixed return value in moveFilesToDerivedTransformation()

*RMS
BUGFIX: CleanReqDBAgent - now() -> utcnow() in initialize()

*Resources
FIX: ARCComputingElement - fix the parsing of CE status if no jobs are available

[v6r9p19]

*DMS
FIX: FileCatalog/DirectoryMetadata - inherited metadata is used while selecting directories
     in findDirIDsByMetadata()

[v6r9p18]

*DMS
FIX: FTSSubmitAgent, FTSRequest - fixes the staging mechanism in the FTS transfer submission
NEW: TransferDBMonitoringHandler - added getFilesForChannel(), resetFileChannelStatus()

[v6r9p17]

*Accounting
FIX: DataStoreClient - send accounting records in batches of 1000 records instead of 100

*DMS:
FIX: FailoverTransfer - catalog name from list to string
FIX: FTSSubmitAgent, FTSRequest - handle FTS3 as new protocol and fix bad submission time
FIX: FTSSubmitAgent, FTSRequest - do not submit FTS transfers for staging files

*WMS
FIX: TaskQueueDB - do not check enabled when TQs are requested from Directors
FIX: TaskQueueDB - check for Enabled in the TaskQueues when inserting jobs to print an alert
NEW: TaskQueueDB - each TQ can have at most 5k jobs, if beyond the limit create a new TQ 
     to prevent long matching times when there are way too many jobs in a single TQ

[v6r9p16]

*TS
BUGFIX: typos in TransformationCleaningAgent.py

*DMS
CHANGE: DownloadInputData - check the available disk space in the right input data directory
FIX: DownloadInputData - try to download only Cached replicas 

[v6r9p15]

*Core
FIX: MySQL - do not decrease the retry counter after ping failure

*DMS
CHANGE: FC/DirectoryMetadata - Speed up findFilesByMetadataWeb when many files match
FIX: RemovalTask - fix error string when removing a non existing file (was incompatible 
     with the LHCb BK client). 

*WMS
FIX: JobReport - minor fix ( removed unused imports )
FIX: JobMonitoring(JobStateUpdate)Handler - jobID argument can be either string, int or long

*TS
CHANGE: TransformationClient - change status of Moved files to a deterministic value
FIX: FileReport - minor fix ( inherits object ) 

[v6r9p14]

*DMS
CHANGE: FTSDB - changed schema: removing FTSSite table. From now on FTS sites 
        would be read from CS Resources

[v6r9p13]

FIX: included fixes from v6r8p26 patch release

[v6r9p12]

FIX: included fixes from v6r8p25 patch release

[v6r9p11]

*DMS
BUGFIX: FTSRequest - in __resolveFTSServer() type "=" -> "=="

[v6r9p10]

FIX: included fixes from v6r8p24 patch release

*Core
NEW: StateMachine utility

*DMS
BUGFIX: in RegisterFile operation handler

*Interfaces
FIX: Dirac.py - in splitInputData() consider only Active replicas

[v6r9p9]

*RMS
FIX: RequestDB - added getRequestFileStatus(), getRequestName() methods

[v6r9p8]

*DMS
FIX: RequestDB - get correct digest ( short request description ) of a request

[v6r9p7]

FIX: included fixes from v6r8p23 patch release

*RSS
FIX: SpaceTokenOccupancyPolicy - SpaceToken Policy decision was based on 
     percentage by mistake
     
*RMS
NEW: new scripts dirac-dms-ftsdb-summary, dirac-dms-show-ftsjobs    
FIX: FTSAgent - setting space tokens for newly created FTSJobs 

[v6r9p6]

*DMS
BUGFIX: dirac-admin-add-ftssite - missing import

*RMS
NEW: RequestDB, ReqManagerHandler - added getRequestStatus() method

*TS
FIX: fixes when using new RequestClient with the TransformationCleaningAgent

*WMS
BUGFIX: typo in SandboxStoreHandler transfer_fromClient() method

[v6r9p5]

*DMS
BUGFIX: missing proxy in service env in the FTSManager service. By default service 
        will use DataManager proxy refreshed every 6 hours.

*Resources
NEW: StorageElement - new checkAccess policy: split the self.checkMethods in 
     self.okMethods. okMethods are the methods that do not use the physical SE. 
     The isValid returns S_OK for all those immediately

*RSS
FIX: SpaceTokenOccupancyPolicy - Policy that now takes into account absolute values 
     for the space left
     
*TS
FIX: TransformationCleaningAgent - will look for both old and new RMS     

[v6r9p4]

*Stager
NEW: Stager API: dirac-stager-monitor-file, dirac-stager-monitor-jobs, 
     dirac-stager-monitor-requests, dirac-stager-show-stats

[v6r9p3]

*Transformation
FIX: TransformationCleaning Agent status was set to 'Deleted' instead of 'Cleaned'

[v6r9p2]

*RSS
NEW: Added Component family tables and statuses
FIX: removed old & unused code 
NEW: allow RSS policies match wild cards on CS

*WMS
BUGFIX: FailoverTransfer,JobWrapper - proper propagation of file metadata

[v6r9p1]

*RMS
NEW: FTSAgent - update rwAccessValidStamp,
     update ftsGraphValidStamp,
     new option for staging files before submission,
     better log handling here and there
CHANGE: FTSJob - add staging flag in in submitFTS2
CHANGE: Changes in WMS (FailoverTransfer, JobReport, JobWrapper, SandboxStoreHandler) 
        and TS (FileReport) to follow the new RMS.
NEW: Full CRUD support in RMS.

*RSS
NEW: ResourceManagementDB - new table ErrorReportBuffer
NEW: new ResourceManagementClient methods - insertErrorReportBuffer, selectErrorReportBuffer,
     deleteErrorReportBuffer

[v6r9]

NEW: Refactored Request Management System, related DMS agents and FTS management
     components

[v6r8p28]

*Core
BUGFIX: RequestHandler - the lock Name includes ActionType/Action

*DMS
FIX: dirac-dms-filecatalog-cli - prevent exception in case of missing proxy

[v6r8p27]

*DMS
BUGFIX: dirac-dms-add-file - fixed typo item -> items

[v6r8p26]

*Core
NEW: RequestHandler - added getServiceOption() to properly resolve inherited options 
     in the global service handler initialize method
NEW: FileCatalogHandler, StorageElementHandler - use getServiceOption()

[v6r8p25]

FIX: included fixes from v6r7p40 patch release

*Resources
FIX: SRM2Storage - do not account gfal_ls operations

[v6r8p24]

FIX: included fixes from v6r7p39 patch release

*Core
FIX: SiteSEMapping was returning wrong info

*DMS
FIX: FTSRequest - choose explicitly target FTS point for RAL and CERN
BUGFIX: StrategyHandler - wrong return value in __getRWAccessForSE()

*Resources
CHANGE: SRM2Storage - do not account gfal_ls operations any more

[v6r8p23]

FIX: included fixes from v6r7p37 patch release

*TS
FIX: TransformationDB - allow tasks made with ProbInFC files
FIX: TransformationCleaingAgent,Client - correct setting of transformation 
     status while cleaning

[v6r8p22]

FIX: included fixes from v6r7p36 patch release

[v6r8p21]

*DMS
FIX: FileCatalog/DirectoryMetadata - even if there is no meta Selection 
     the path should be considered when getting Compatible Metadata
FIX: FileCatalog/DirectoryNodeTree - findDir will return S_OK( '' ) if dir not 
     found, always return the same error from DirectoryMetadata in this case.     

*RSS
FIX: DowntimeCommand - use UTC time stamps

*TS
FIX: TransformationAgent - in _getTransformationFiles() get also ProbInFC files in 
     addition to Used 

[v6r8p20]

*Stager
NEW: Stager API: dirac-stager-monitor-file, dirac-stager-monitor-jobs, 
     dirac-stager-monitor-requests, dirac-stager-show-stats

[v6r8p19]

*Transformation
FIX: TransformationCleaning Agent status was set to 'Deleted' instead of 'Cleaned'

[v6r8p18]

*TS
BUGFIX: TransformationAgent - regression in __cleanCache()

[v6r8p17]

FIX: included fixes from v6r7p32 patch release

*WMS
FIX: StalledJobAgent - for accidentally stopped jobs ExecTime can be not set, 
     set it to CPUTime for the accounting purposes in this case

[v6r8p16]

FIX: included fixes from v6r7p31 patch release

*WMS
BUGFIX: TaskQueueDB - fixed a bug in the negative matching conditions SQL construction

*RSS
NEW: improved doc strings of PEP, PDP modules ( part of PolicySystem )
FIX: Minor changes to ensure consistency if ElementInspectorAgent and 
     users interact simultaneously with the same element
CHANGE: removed DatabaseCleanerAgent ( to be uninstalled if already installed )
FIX: SummarizeLogsAgent - the logic of the agent was wrong, the agent has been re-written.
     
[v6r8p15]

*Core
FIX: X509Chain - fix invalid information when doing dirac-proxy-info without CS
     ( in getCredentials() )

*RSS
NEW: PDP, PEP - added support for option "doNotCombineResult" on PDP

[v6r8p14]

*Core
FIX: dirac-deploy-scripts - can now work with the system python

*WMS
NEW: dirac-wms-cpu-normalization - added -R option to modify a given configuration file
FIX: Executor/InputData - Add extra check for LFns in InputData optimizer, closes #1472

*Transformation
CHANGE: TransformationAgent - add possibility to kick a transformation (not skip it if no 
        unused files), by touching a file in workDirectory
BUGFIX: TransformationAgent - bug in __cleanCache() dict modified in a loop        

[v6r8p13]

*Transformation
BUGFIX: TransformationDB - restored import of StringType

[v6r8p12]

NEW: Applied patches from v6r7p29

*WMS
FIX: JobDB - check if SystemConfig is present in the job definition and convert it 
     into Platform

*DMS
FIX: ReplicaManager - do not get metadata of files when getting files in a directory 
     if not strictly necessary

*RSS
NEW: ported from LHCb PublisherHandler for RSS web views

[v6r8p11]

NEW: Applied patches from v6r7p27

*RSS
NEW: SpaceTokenOccupancyPolicy - ported from LHCbDIRAC 
NEW: db._checkTable done on service initialization ( removed dirac-rss-setup script doing it )

*Transformation
FIX: TaskManager - reset oJob for each task in prepareTransformationTasks()
BUGFIX: ValidateOutputDataAgent - typo fixed in getTransformationDirectories()
FIX: TransformationManagerHandler - use CS to get files statuses not to include in 
     processed file fraction calculation for the web monitoring pages

[v6r8p10]

NEW: Applied patches from v6r7p27

[v6r8p9]

*DMS
FIX: TransferAgent,dirac-dms-show-se-status, ResourceStatus,TaskManager - fixes
     needed for DMS components to use RSS status information
NEW: ReplicaManager - allow to get metadata for an LFN+SE as well as PFN+SE     

[v6r8p8]

*RSS
BUGFIX: dirac-rss-setup - added missing return of S_OK() result

[v6r8p7]

NEW: Applied patches from v6r7p24

*DMS
BUGFIX: LcgFileCatalogClient - bug in addFile()

*RSS
BUGFIX: fixed script dirac-rss-set-token, broken in the current release.
NEW: Statistics module - will be used in the future to provide detailed information 
     from the History of the elements 

[v6r8p6]

NEW: Applied patches from v6r7p23

*Transformation
FIX: TaskManager - allow prepareTransformationTasks to proceed if no OutputDataModule is defined
FIX: TransformationDB - remove INDEX(TaskID) from TransformationTasks. It produces a single counter 
     for the whole table instead of one per TransformationID
     
*WMS     
FIX: WMSUtilities - to allow support for EMI UI's for pilot submission we drop support for glite 3.1

[v6r8p5]

NEW: Applied patches from v6r7p22

*RSS
CHANGE: removed old tests and commented out files

*WMS
FIX: PoolXMLCatalog - proper addFile usage

*Transformation
CHANGE: TransformationAgent - clear replica cache when flushing or setting a file in the workdirectory

[v6r8p4]

*Transformation
FIX: The connection to the jobManager is done only at submission time
FIX: Jenkins complaints fixes

*WMS
BUGFIX: JobDB - CPUtime -> CPUTime
FIX: Jenkins complaints fixes

[v6r8p3]

*DMS
BUGFIX: LcgFileCatalogClient

[v6r8p2]

*DMS:
FIX: LcgFileCatalogClient - remove check for opening a session in __init__ as credentials are not yet set 

*Transformation
CHANGE: reuse RPC clients in Transformation System 

[v6r8p1]

*Core
FIX: dirac-deploy-scripts - restored regression w.r.t. support of scripts starting with "d"

*DMS
BUGFIX: LcgFileCatalogClient - two typos fixed

[v6r8]

CHANGE: Several fixes backported from the v7r0 integration branch

*Core
CHANGE: DictCache - uses global LockRing to avoid locks in multiprocessing
FIX: X509Chain - proxy-info showing an error when there's no CS

*DMS
FIX: TransferAgent - inside loop filter out waiting files dictionary
BUGFIX: dirac-admin-allow-se - there was a continue that was skipping the complete loop for 
        ARCHIVE elements
NEW: LcgFileCatalogClient - test return code in startsess lfc calls       

*WMS:
FIX: OptimizerExecutor, InputData, JobScheduling - check that site candidates have all the 
     replicas

*RSS: 
BUGFIX: ResourceStatus, RSSCacheNoThread - ensure that locks are always released

*Transformation
FIX: TaskManager - site in the job definition is taken into account when submitting
NEW: Transformation - get the allowed plugins from the CS /Operations/Transformations/AllowedPlugins
FIX: ValidateOutputDataAgent - self not needed for static methods

[v6r7p40]

*Resources
FIX: StorageElement class was not properly passing the lifetime argument for prestageFile method

[v6r7p39]

*Core
CHANGE: Grid - in executeGridCommand() allow environment script with arguments needed for ARC client

*DMS
FIX: DFC SEManager - DIP Storage can have a list of ports now

*Resources
FIX: ARCComputingElement - few fixes after debugging

[v6r7p38]

*Core
NEW: DISET FileHelper, TransferClient - possibility to switch off check sum

*Resources
NEW: ARCComputingElement - first version
NEW: StorageFactory - possibility to pass extra protocol parameters to storage object
NEW: DIPStorage - added CheckSum configuration option
BUGFIX: SSHComputingElement - use CE name in the pilot reference construction

*WMS
FIX: StalledJobAgent - if ExecTime < CPUTime make it equal to CPUTime

[v6r7p37]

*Framework
BUGFIX: NotificationDB - typos in SQL statement in purgeExpiredNotifications() 

*WMS
NEW: JobCleaningAgent - added scheduling sandbox LFN removal request 
     when deleting jobs
CHANGE: JobWrapper - report only error code as ApplicationError parameter 
        when payload finishes with errors    
NEW: SiteDirector - possibility to specify extensions to be installed in 
     pilots in /Operations/Pilots/Extensions option in order not to install
     all the server side extensions        

*DMS
CHANGE: FileCatalogFactory - use service path as default URL
CHANGE: FileCatalogFactory - use ObjectLoader to import catalog clients

*SMS
BUGFIX: StorageManagementDB, dirac-stager-monitor-jobs - small bug fixes ( sic, Daniela )

*Resources
CHANGE: DIPStorage - added possibility to specify a list of ports for multiple
        service end-points
CHANGE: InProcessComputingElement - demote log message when payload failure 
        to warning, the job will fail anyway
FIX: StalledJobAgent - if pilot reference is not registered, this is not an 
     error of the StalledJobAgent, no log.error() in  this case                
        
*RMS
CHANGE: RequestTask - ensure that tasks are executed with user credentials 
        even with respect to queries to DIRAC services ( useServerCertificate 
        flag set to false )        

[v6r7p36]

*WMS
FIX: CREAMCE, SiteDirector - make sure that the tmp executable is removed
CHANGE: JobWrapper - remove sending mails via Notification Service in case
        of job rescheduling
        
*SMS
FIX: StorageManagementDB - fix a race condition when old tasks are set failed 
     between stage submission and update.        

[v6r7p35]

*Stager
NEW: Stager API: dirac-stager-monitor-file, dirac-stager-monitor-jobs, 
     dirac-stager-monitor-requests, dirac-stager-show-stats

[v6r7p34]

*Transformation
FIX: TransformationCleaning Agent status was set to 'Deleted' instead of 'Cleaned'

[v6r7p33]

*Interfaces
FIX: Job.py - in setExecutable() - prevent changing the log file name string type

*StorageManagement
NEW: StorageManagementDB(Handler) - kill staging requests at the same time as 
     killing related jobs, closes #1510
FIX: StorageManagementDB - demote the level of several log messages       

[v6r7p32]

*DMS
FIX: StorageElementHandler - do not use getDiskSpace utility, use os.statvfs instead
CHANGE: StorageManagementDB - in getStageRequests() make MySQL do an UNIQUE selection 
        and use implicit loop to speed up queries for large results

*Resources
FIX: lsfce remote script - use re.search instead of re.match in submitJob() to cope with
     multipline output

[v6r7p31]

*WMS
FIX: SiteDirector - make possible more than one SiteDirector (with different pilot identity) attached 
     to a CE, ie sgm and pilot roles. Otherwise one is declaring Aborted the pilots from the other.

[v6r7p30]

*Core
CHANGE: X509Chain - added groupProperties field to the getCredentials() report
BUGFIX: InstallTools - in getSetupComponents() typo fixed: agent -> executor

[v6r7p29]

*DMS
CHANGE: FileCatalog - selection metadata is also returned as compatible metadata in the result
        of getCompatibleMetadata() call
NEW: FileCatalog - added path argument to getCompatibleMetadata() call
NEW: FileCatalogClient - added getFileUserMetadata()
BUGFIX: dirac-dms-fts-monitor - exit with code -1 in case of error

*Resources
FIX: CREAMComputingElement - check globus-url-copy result for errors when retrieving job output

[v6r7p28]

*DMS
BUGFIX: FileCatalog/DirectoryMetadata - wrong MySQL syntax 

[v6r7p27]

*Core
FIX: Mail.py - fix of the problem of colons in the mail's body

*Interfaces
NEW: Job API - added setSubmitPools(), setPlatform() sets ... "Platform"

*WMS
FIX: TaskQueueDB - use SystemConfig as Platform for matching ( if Platform is not set explicitly

*Resources
FIX: SSHComputingElement - use ssh host ( and not CE name ) in the pilot reference
BUGFIX: SSHGEComputingElement - forgotten return statement in _getJobOutputFiles()

*Framework
NEW: dirac-sys-sendmail - email's body can be taken from pipe. Command's argument 
     in this case will be interpreted as a destination address     

[v6r7p26]

*DMS
FIX: ReplicaManager - status names Read/Write -> ReadAccess/WriteAccess

[v6r7p25]

*Core
CHANGE: X509Chain - in getCredentials() failure to contact CS is not fatal, 
        can happen when calling dirac-proxy-init -x, for example

[v6r7p24]

*DMS
NEW: FileCatalog - added getFilesByMetadataWeb() to allow pagination in the Web 
     catalog browser
     
*WMS
CHANGE: WMSAdministrator, DiracAdmin - get banned sites list by specifying the status
        to the respective jobDB call     

[v6r7p23]

*Transformation
BUGFIX: TransformationDB - badly formatted error log message

*RMS
CHANGE: RequestDBMySQL - speedup the lookup of requests

*WMS
BUGFIX: dirac-dms-job-delete - in job selection by group

*DMS
FIX: LcgFileCatalogClient - getDirectorySize made compatible with DFC
BUGFIX: LcgFileCatalogClient - proper call of __getClientCertInfo()

[v6r7p22]

*Transformation
CHANGE: InputDataAgent - treats only suitable transformations, e.g. not the extendable ones. 
CHANGE: TransformationAgent - make some methods more public for easy overload

[v6r7p21]

*Core
FIX: Shifter - pass filePath argument when downloading proxy

[v6r7p20]

*DMS
CHANGE: StrategyHandler - move out SourceSE checking to TransferAgent
CHANGE: ReplicaManager, InputDataAgent - get active replicas
FIX: StorageElement, SRM2Storage - support for '*Access' statuses, checking results
     of return structures
     
*RSS
NEW: set configurable email address on the CS to send the RSS emails
NEW: RSSCache without thread in background
FIX: Synchronizer - moved to ResourceManager handler     

[v6r7p19]

*DMS
BUGFIX: ReplicaManager - in putAndRegister() SE.putFile() singleFile argument not used explicitly

[v6r7p18]

*WMS
FIX: StalledJobAgent - do not exit the loop over Completed jobs if accounting sending fails
NEW: dirac-wms-job-delete - allow to specify jobs to delete by job group and/or in a file
FIX: JobManifest - If CPUTime is not set, set it to MaxCPUTime value

[v6r7p17]

*Resources
FIX: SRM2Storage - treat properly "22 SRM_REQUEST_QUEUED" result code

[v6r7p16]

*DMS
FIX: StrategyHandler - do not proceed when the source SE is not valid for read 
BUGFIX: StorageElement - putFile can take an optional sourceSize argument
BUGFIX: ReplicaManager - in removeFile() proper loop on failed replicas

*RSS
FIX: SpaceTokenOccupancyCommand, CacheFeederAgent - add timeout when calling lcg_util commands

*WMS
FIX: JobManifest - take all the SubmitPools defined in the TaskQueueAgent 
NEW: StalledJobAgent - declare jobs stuck in Completed status as Failed

[v6r7p15]

*Core
BUGFIX: SocketInfo - in host identity evaluation

*DMS
BUGFIX: FileCatalogHandler - missing import os

*Transformation
CHANGE: JobManifest - getting allowed job types from operations() section 

[v6r7p14]

*DMS
CHANGE: StorageElementProxy - removed getParameters(), closes #1280
FIX: StorageElementProxy - free the getFile space before the next file
FIX: StorageElement - added getPFNBase() to comply with the interface

*Interfaces
CHANGE: Dirac API - allow lists of LFNs in removeFile() and removeReplica()

*WMS
CHANGE: JobSchedulingAgent(Executor) - allow both BannedSite and BannedSites JDL option

*RSS
FIX: ElementInspectorAgent - should only pick elements with rss token ( rs_svc ).
FIX: TokenAgent - using 4th element instead of the 5th. Added option to set admin email on the CS.

[v6r7p13]

*Core
FIX: Resources - in getStorageElementSiteMapping() return only sites with non-empty list of SEs

*DMS
FIX: StorageElement - restored the dropped logic of using proxy SEs
FIX: FileCatalog - fix the UseProxy /LocalSite/Catalog option

*Transformation
FIX: TransformationDB - use lower() string comparison in extendTransformation()

[v6r7p12]

*WMS
BUGFIX: JobManifest - get AllowedSubmitPools from the /Systems section, not from /Operations

*Core
NEW: Resources helper - added getSites(), getStorageElementSiteMapping()

*DMS
CHANGE: StrategyHandler - use getStorageElementSiteMapping helper function
BUGFIX: ReplicaManager - do not modify the loop dictionary inside the loop

[v6r7p11]

*Core
CHANGE: Subprocess - put the use of watchdog in flagging

[v6r7p10]

*Core
NEW: Logger - added getLevel() method, closes #1292
FIX: Subprocess - returns correct structure in case of timeout, closes #1295, #1294
CHANGE: TimeOutExec - dropped unused utility
FIX: Logger - cleaned unused imports

*RSS
CHANGE: ElementInspectorAgent - do not use mangled name and removed shifterProxy agentOption

[v6r7p9]

*Core
BUGFIX: InstallTools - MySQL Port should be an integer

[v6r7p8]

*Core
FIX: Subprocess - consistent timeout error message

*DMS
NEW: RemovalTask - added bulk removal
FIX: StrategyHandler - check file source CEs
CHANGE: DataIntegrityClient - code beautification
CHANGE: ReplicaManager - do not check file existence if replica information is queried anyway,
        do not fail if file to be removed does not exist already. 

[v6r7p7]

FIX: Several fixes to allow automatic code documentation

*Core
NEW: InstallTools - added mysqlPort and mysqlRootUser

*DMS
CHANGE: ReplicaManager - set possibility to force the deletion of non existing files
CHANGE: StrategyHandler - better handling of checksum check during scheduling 

[v6r7p6]

*Core
FIX: dirac-install - restore signal alarm if downloadable file is not found
FIX: Subprocess - using Manager proxy object to pass results from the working process

*DMS:
CHANGE: StorageElement - removed overwride mode
CHANGE: removed obsoleted dirac-dms-remove-lfn-replica, dirac-dms-remove-lfn
NEW: FTSMonitorAgent - filter out sources with checksum mismatch
FIX: FTSMonitorAgent, TransferAgent - fix the names of the RSS states

*RSS
NEW: ElementInspectorAgent runs with a variable number of threads which are automatically adjusted
NEW: Added policies to force a particular state, can be very convenient to keep something Banned for example.
NEW: policy system upgrade, added finer granularity when setting policies and actions

*WMS
NEW: SiteDirector- allow to define pilot DN/Group in the agent options
CHANGE: JobDescription, JobManifest - take values for job parameter verification from Operations CS section

[v6r7p5]

*Interfaces
BUGFIX: dirac-wms-job-get-output - properly treat the case when output directory is not specified 

[v6r7p4]

*Core
FIX: Subprocess - avoid that watchdog kills the executor process before it returns itself

*Framework
BUGFIX: ProxuManagerClient - wrong time for caching proxies

*RSS
FIX: removed obsoleted methods

*DMS
NEW: FileCatalog - added findFilesByMetadataDetailed - provides detailed metadata for 
     selected files

[v6r7p3]

*DMS
FIX: FTSMonitorAgent - logging less verbose

*Transformation
FIX: TransformationAgent - use the new CS defaults locations
FIX: Proper agent initialization
NEW: TransformationPlaugin - in Broadcast plugin added file groupings by number of files, 
     make the TargetSE always defined, even if the SourceSE list contains it 

*ResourceStatus
FIX: Added the shifter's proxy to several agents

*RMS
FIX: RequestContainer - the execution order was not properly set for the single files 

*Framework:
BUGFIX: ProxyManagerClient - proxy time can not be shorter than what was requested

[v6r7p2]

*Core
FIX: dirac-configure - switch to use CS before checking proxy info

*Framework
NEW: dirac-sys-sendmail new command
NEW: SystemAdmininistratorCLI - added show host, uninstall, revert commands
NEW: SystemAdmininistratorHandler - added more info in getHostInfo()
NEW: SystemAdmininistratorHandler - added revertSoftware() interface

*Transformation
FIX: TransformationCleaningAgent - check the status of returned results

[v6r7p1]

*Core
FIX: Subprocess - finalize the Watchdog closing internal connections after a command execution
CHANGE: add timeout for py(shell,system)Call calls where appropriate
CHANGE: Shifter - use gProxyManager in a way that allows proxy caching

*Framework
NEW: ProxyManagerClient - allow to specify validity and caching time separately
FIX: ProxyDB - replace instead of delete+insert proxy in __storeVOMSProxy

*DMS
NEW: FTSMonitorAgent - made multithreaded for better efficiency
FIX: dirac-dms-add-file - allow LFN: prefix for lfn argument

*WMS
NEW: dirac-wms-job-get-output, dirac-wms-job-status - allow to retrieve output for a job group
FIX: TaskQueueDB - fixed selection SQL in __generateTQMatchSQL()
CHANGE: OptimizerExecutor - reduce diversity of MinorStatuses for failed executors

*Resources
FIX: CREAMComputingElement - remove temporary JDL right after the submission 

[v6r6p21]

*DMS
BUGFIX: TransformationCleaningAgent - use the right signature of cleanMetadataCatalogFiles() call

[v6r6p20]

*DMS
FIX: RegistrationTask - properly escaped error messages
BUGFIX: DirectoryMetadata - use getFileMetadataFields from FileMetadata in addMetadataField()
NEW: When there is a missing source error spotted during FTS transfer, file should be reset 
     and rescheduled again until maxAttempt (set to 100) is reached

*WMS
FIX: JobScheduling - fix the site group logic in case of Tier0

[v6r6p19]

*DMS
BUGFIX: All DMS agents  - set up agent name in the initialization

*Core
NEW: Subprocess - timeout wrapper for subprocess calls
BUGFIX: Time - proper interpreting of 0's instead of None
CHANGE: DISET - use cStringIO for ANY read that's longer than 16k (speed improvement) 
        + Less mem when writing data to the net
FIX: Os.py - protection against failed "df" command execution       
NEW: dirac-info prints lcg bindings versions
CHANGE: PlotBase - made a new style class 
NEW: Subprocess - added debug level log message

*Framework
NEW: SystemAdministratorIntegrator client for collecting info from several hosts
NEW: SystemAdministrator - added getHostInfo()
FIX: dirac-proxy-init - always check for errors in S_OK/ERROR returned structures
CHANGE: Do not accept VOMS proxies when uploading a proxy to the proxy manager

*Configuration
FIX: CE2CSAgent - get a fresh copy of the cs data before attempting to modify it, closes #1151
FIX: Do not create useless backups due to slaves connecting and disconnecting
FIX: Refresher - prevent retrying with 'Insane environment'

*Accounting
NEW: Accounting/Job - added validation of reported values to cope with the weird Yandex case
FIX: DBUtils - take into account invalid values, closes #949

*DMS
FIX: FTSSubmitAgent - file for some reason rejected from submission should stay in 'Waiting' in 
     TransferDB.Channel table
FIX: FTSRequest - fix in the log printout     
CHANGE: dirac-dms-add-file removed, dirac-dms-add-files renamed to dirac-dms-add-file
FIX: FileCatalogCLI - check the result of removeFile call
FIX: LcgFileCatalogClient - get rid of LHCb specific VO evaluation
NEW: New FileCatalogProxy service - a generalization of a deprecated LcgFileCatalog service
FIX: Restored StorageElementProxy functionality
CHANGE: dirac-dms-add-file - added printout
NEW: FileCatalog(Factory), StorageElement(Factory) - UseProxy flag moved to /Operations and /LocalSite sections

*RSS
NEW:  general reimplementation: 
      New DB schema using python definition of tables, having three big blocks: Site, Resource and Node.
      MySQLMonkey functionality almost fully covered by DB module, eventually will disappear.
      Services updated to use new database.
      Clients updated to use new database.
      Synchronizer updated to fill the new database. When helpers will be ready, it will need an update.
      One ElementInspectorAgent, configurable now is hardcoded.
      New Generic StateMachine using OOP.
      Commands and Policies simplified.
      ResourceStatus using internal cache, needs to be tested with real load.
      Fixes for the state machine
      Replaced Bad with Degraded status ( outside RSS ).
      Added "Access" to Read|Write|Check|Remove SE statuses wherever it applies.
      ResourceStatus returns by default "Active" instead of "Allowed" for CS calls.
      Caching parameters are defined in the CS
FIX: dirac-admin-allow/ban-se - allow a SE on Degraded ( Degraded->Active ) and ban a SE on Probing 
     ( Probing -> Banned ). In practice, Active and Degraded are "usable" states anyway.            
      
*WMS
FIX: OptimizerExecutor - failed optimizations will still update the job     
NEW: JobWrapper - added LFNUserPrefix VO specific Operations option used for building user LFNs
CHANGE: JobDB - do not interpret SystemConfig in the WMS/JobDB
CHANGE: JobDB - Use CPUTime JDL only, keep MaxCPUTime for backward compatibility
CHANGE: JobWrapper - use CPUTime job parameter instead of MaxCPUTime
CHANGE: JobAgent - use CEType option instead of CEUniqueID
FIX: JobWrapper - do not attempt to untar directories before having checked if they are tarfiles 
NEW: dirac-wms-job-status - get job statuses for jobs in a given job group
 
*SMS
FIX: StorageManagementDB - when removing unlinked replicas, take into account the case where a
     staging request had been submitted, but failed
      
*Resources    
NEW: glexecCE - add new possible locations of the glexec binary: OSG specific stuff and in last resort 
     looking in the PATH    
NEW: LcgFileCatalogClient - in removeReplica() get the needed PFN inside instead of providing it as an argument     
      
*TS      
CHANGE: Transformation types definition are moved to the Operations CS section

*Interfaces
FIX: Dirac.py - CS option Scratchdir was in LocalSite/LocalSite
FIX: Dirac.py - do not define default catalog, use FileCatalog utility instead

[v6r6p19]

*DMS
BUGFIX: All DMS agents  - set up agent name in the initialization

[v6r6p18]

*Transformation
CHANGE: /DIRAC/VOPolicy/OutputDataModule option moved to <Operations>/Transformations/OutputDataModule

*Resources
FIX: ComputingElement - properly check if the pilot proxy has VOMS before adding it to the payload 
     when updating it

*WMS
BUGFIX: JobSanity - fixed misspelled method call SetParam -> SetParameter

[v6r6p17]

*Transformation
BUGFIX: TransformationAgent - corrected  __getDataReplicasRM()

[v6r6p16]

*DMS
FIX: Agents - proper __init__ implementation with arguments passing to the super class
FIX: LcgFileCatalogClient - in removeReplica() reload PFN in case it has changed

[v6r6p15]

*Framework
BUGFIX: ErrorMessageMonitor - corrected updateFields call 

*DMS:
NEW: FTSMonitorAgent completely rewritten in a multithreaded way

*Transformation
FIX: InputDataAgent - proper instantiation of TransformationClient
CHANGE: Transformation - several log message promoted from info to notice level

[v6r6p14]

*Transformation
FIX: Correct instantiation of agents inside several scripts
CHANGE: TransformationCleaningAgent - added verbosity to logs
CHANGE: TransformationAgent - missingLFC to MissingInFC as it could be the DFC as well
FIX: TransformationAgent - return an entry for all LFNs in __getDataReplicasRM

*DMS
FIX: TransferAgent - fix exception reason in registerFiles()

[v6r6p13]

*DMS
CHANGE: TransferAgent - change RM call from getCatalogueReplicas to getActiveReplicas. 
        Lowering log printouts here and there

[v6r6p12]

*DMS
BUGFIX: RemovalTask - Replacing "'" by "" in error str set as attribute for a subRequest file. 
        Without that request cannot be updated when some nasty error occurs.

[v6r6p11]

*RMS:
BUGFIX: RequestClient - log string formatting

*DMS
BUGFIX: RemovalTask - handling for files not existing in the catalogue

*Transformation
FIX: TransformationManager - ignore files in NotProcessed status to get the % of processed files

*Interfaces
FIX: Fixes due to the recent changes in PromptUser utility

[v6r6p10]

*RMS
FIX: RequestDBMySQL - better escaping of queries 

*WMS
FIX: SiteDirector - get compatible platforms before checking Task Queues for a site

[v6r6p9]

*Core
FIX: Utilities/PromptUser.py - better user prompt

*Accounting
NEW: Add some validation to the job records because of weird data coming from YANDEX.ru

*DMS
BUGFIX: ReplicaManager - typo errStr -> infoStr in __replicate()
FIX: FTSRequest - fixed log message

*WMS
FIX: SiteDirector - use CSGlobals.getVO() call instead of explicit CS option

[v6r6p8]

*Transformation
BUGFIX: TransformationDB - typo in getTransformationFiles(): iterValues -> itervalues

[v6r6p7]

*Resources
FIX: StorageFactory - uncommented line that was preventing the status to be returned 
BUGFIX: CE remote scripts - should return status and not call exit()
BUGFIX: SSHComputingElement - wrong pilot ID reference

[v6r6p6]

*WMS
FIX: TaskQueueDB - in findOrphanJobs() retrieve orphaned jobs as list of ints instead of list of tuples
FIX: OptimizerExecutor - added import of datetime to cope with the old style optimizer parameters

*Transformation
FIX: TransformationAgent - fix finalization entering in an infinite loop
NEW: TransformationCLI - added resetProcessedFile command
FIX: TransformationCleaningAgent - treating the archiving delay 
FIX: TransformationDB - fix in getTransformationFiles() in case of empty file list

[v6r6p5]

*Transformation
FIX: TransformationAgent - type( transClient -> transfClient )
FIX: TransformationAgent - self._logInfo -> self.log.info
FIX: TransformationAgent - skip if no Unused files
FIX: TransformationAgent - Use CS option for replica cache lifetime
CHANGE: TransformationAgent - accept No new Unused files every [6] hours

[v6r6p4]

*DMS
FIX: TransferAgent - protection for files that can not be scheduled
BUGFIX: TransferDB - typo (instIDList - > idList ) fixed

*Transformation
BUGFIX: TransformationAgent - typo ( loginfo -> logInfo )

[v6r6p3]

FIX: merged in patch v6r5p14

*Core
BUGFIX: X509Chain - return the right structure in getCredentials() in case of failure
FIX: dirac-deploy-scripts.py - allow short scripts starting from "d"
FIX: dirac-deploy-scripts.py - added DCOMMANDS_PPID env variable in the script wrapper
FIX: ExecutorReactor - reduced error message dropping redundant Task ID 

*Interfaces
BUGFIX: Dirac.py - allow to pass LFN list to replicateFile()

*DMS
FIX: FileManager - extra check if all files are available in _findFiles()
BUGFIX: FileCatalogClientCLI - bug in DirectoryListing

[v6r6p2]

FIX: merged in patch v6r5p13

*WMS
FIX: SiteDirector - if no community set, look for DIRAC/VirtualOrganization setting

*Framework
FIX: SystemLoggingDB - LogLevel made VARCHAR in the MessageRepository table
FIX: Logging - several log messages are split in fixed and variable parts
FIX: SystemLoggingDB - in insertMessage() do not insert new records in auxiliary tables if they 
     are already there

[v6r6p1]

*Core:
CHANGE: PromptUser - changed log level of the printout to NOTICE
NEW: Base Client constructor arguments are passed to the RPCClient constructor

*DMS:
NEW: FTSRequest - added a prestage mechanism for source files
NEW: FileCatalogClientCLI - added -f switch to the size command to use raw faile tables 
     instead of storage usage tables
NEW: FileCatalog - added orphan directory repair tool
NEW: FIleCatalog - more counters to control the catalog sanity     

*WMS:
FIX: SandboxStoreClient - no more kwargs tricks
FIX: SandboxStoreClient returns sandbox file name in case of upload failure to allow failover
FIX: dirac-pilot - fixed VO_%s_SW_DIR env variable in case of OSG

*TS:
FIX: TransformationManagerHandler - avoid multiple Operations() instantiation in 
     getTransformationSummaryWeb()

[v6r6]

*Core
CHANGE: getDNForUsername helper migrated from Core.Security.CS to Registry helper
NEW: SiteSEMapping - new utilities getSitesGroupedByTierLevel(), getTier1WithAttachedTier2(),
     getTier1WithTier2
CHANGE: The DIRAC.Core.Security.CS is replaced by the Registry helper     
BUGFIX: dirac-install - properly parse += in .cfg files
FIX: Graphs.Utilities - allow two lines input in makeDataFromCVS()
FIX: Graphs - allow Graphs package usage if even matplotlib is not installed
NEW: dirac-compile-externals will retrieve the Externals compilation scripts from it's new location 
     in github (DIRACGrid/Externals)
NEW: Possibility to define a thread-global credentials for DISET connections (for web framework)
NEW: Logger - color output ( configurable )
NEW: dirac-admin-sort-cs-sites - to sort sites in the CS
CHANGE: MessageClient(Factor) - added msgClient attribute to messages
NEW: Core.Security.Properties - added JOB_MONITOR and USER_MANAGER properties

*Configuration
NEW: Registry - added getAllGroups() method

*Framework
NEW: SystemAdministratorClientCLI - possibility to define roothPath and lcgVersion when updating software

*Accounting
NEW: JobPlotter - added Normalized CPU plots to Job accounting
FIX: DBUtils - plots going to greater granularity

*DMS
NEW: FileCatalog - storage usage info stored in all the directories, not only those with files
NEW: FileCatalog - added utility to rebuild storage usage info from scratch
FIX: FileCatalog - addMetadataField() allow generic types, e.g. string
FIX: FileCatalog - path argument is normalized before usage in multiple methods
FIX: FileCatalog - new metadata for files(directories) should not be there before for directories(files)
NEW: FileCatalog - added method for rebuilding DirectoryUsage data from scratch 
NEW: FileCatalog - Use DirectoryUsage mechanism for both logical and physical storage
CHANGE: FileCatalog - forbid removing non-empty directories
BUGFIX: FileCatalogClientCLI - in do_ls() check properly the path existence
FIX: FileCatalogClientCLI - protection against non-existing getCatalogCounters method in the LFC client
FIX: DMS Agents - properly call superclass constructor with loadName argument
FIX: ReplicaManager - in removeFile() non-existent file is marked as failed
FIX: Make several classes pylint compliant: DataIntegrityHandler, DataLoggingHandler,
     FileCatalogHandler, StorageElementHandler, StorageElementProxyHandler, TransferDBMonitoringHandler
FIX: LogUploadAgent - remove the OSError exception in __replicate()
FIX: FileCatalogClientCLI - multiple check of proper command inputs,
     automatic completion of several commands with subcommands,
     automatic completion of file names
CHANGE: FileCatalogClientCLI - reformat the output of size command 
FIX: dirac-admin-ban-se - allow to go over all options read/write/check for each SE      
NEW: StrategyHandler - new implementation to speed up file scheduling + better error reporting
NEW: LcgFileCatalogProxy - moved from from LHCbDirac to DIRAC
FIX: ReplicaManager - removed usage of obsolete "/Resources/StorageElements/BannedTarget" 
CHANGE: removed StorageUsageClient.py
CHANGE: removed obsoleted ProcessingDBAgent.py

*WMS
CHANGE: RunNumber job parameter was removed from all the relevant places ( JDL, JobDB, etc )
NEW: dirac-pilot - add environment setting for SSH and BOINC CEs
NEW: WMSAdministrator - get output for non-grid CEs if not yet in the DB
NEW: JobAgent - job publishes BOINC parameters if any
CHANGE: Get rid of LHCbPlatform everywhere except TaskQueueDB
FIX: SiteDirector - provide list of sites to the Matcher in the initial query
FIX: SiteDirector - present a list of all groups of a community to match TQs
CHANGE: dirac-boinc-pilot dropped
CHANGE: TaskQueueDirector does not depend on /LocalSite section any more
CHANGE: reduced default delays for JobCleaningAgent
CHANGE: limit the number of jobs received by JobCleaningAgent
CHANGE: JobDB - use insertFields instead of _insert
CHANGE: Matcher, TaskQueueDB - switch to use Platform rather than LHCbPlatform retaining LHCbPlatform compatibility
BUGFIX: Matcher - proper reporting pilot site and CE
CHANGE: JobManager - improved job Killing/Deleting logic
CHANGE: dirac-pilot - treat the OSG case when jobs on the same WN all run in the same directory
NEW: JobWrapper - added more status reports on different failures
FIX: PilotStatusAgent - use getPilotProxyFromDIRACGroup() instead of getPilotProxyFromVOMSGroup()
CHANGE: JobMonitoringHandler - add cutDate and condDict parameters to getJobGroup()
NEW: JobMonitoringHandler - check access rights with JobPolicy when accessing job info from the web
NEW: JobManager,JobWrapper - report to accounting jobs in Rescheduled final state if rescheduling is successful
FIX: WMSAdministrator, SiteDirector - store only non-empty pilot output to the PilotDB
NEW: added killPilot() to the WMSAdministrator interface, DiracAdmin and dirac-admin-kill-pilot command
NEW: TimeLeft - renormalize time left using DIRAC Normalization if available
FIX: JobManager - reconnect to the OptimizationMind in background if not yet connected
CHANGE: JobManifest - use Operations helper
NEW: JobCleaningAgent - delete logging records from JobLoggingDB when deleting jobs

*RMS
FIX: RequestDBFile - better exception handling in case no JobID supplied
FIX: RequestManagerHandler - make it pylint compliant
NEW: RequestProxyHandler - is forwarding requests from voboxes to central RequestManager. 
     If central RequestManager is down, requests are dumped into file cache and a separate thread 
     running in background is trying to push them into the central. 
CHANGE: Major revision of the code      
CHANGE: RequestDB - added index on SubRequestID in the Files table
CHANGE: RequestClient - readRequestForJobs updated to the new RequetsClient structure

*RSS
NEW: CS.py - Space Tokens were hardcoded, now are obtained after scanning the StorageElements.

*Resources
FIX: SSHComputingElement - enabled multiple hosts in one queue, more debugging
CHANGE: SSHXXX Computing Elements - define SSH class once in the SSHComputingElement
NEW: SSHComputingElement - added option to define private key location
CHANGE: Get rid of legacy methods in ComputingElement
NEW: enable definition of ChecksumType per SE
NEW: SSHBatch, SSHCondor Computing Elements
NEW: SSHxxx Computing Elements - using remote control scripts to better capture remote command errors
CHANGE: put common functionality into SSHComputingElement base class for all SSHxxx CEs
NEW: added killJob() method tp all the CEs
NEW: FileCatalog - take the catalog information info from /Operations CS section, if defined there, 
     to allow specifications per VO 

*Interfaces
CHANGE: Removed Script.initialize() from the API initialization
CHANGE: Some general API polishing
FIX: Dirac.py - when running in mode="local" any directory in the ISB would not get untarred, 
     contrary to what is done in the JobWrapper

*TS
BUGFIX: TaskManager - bug fixed in treating tasks with input data
FIX: TransformationCleaningAgent - properly call superclass constructor with loadName argument
NEW: TransformationCleaningAgent - added _addExtraDirectories() method to extend the list of
     directories to clean in a subclass if needed
CHANGE: TransformationCleaningAgent - removed usage of StorageUsageClient     
NEW: TransformationAgent is multithreaded now ( implementation moved from LHCbDIRAC )
NEW: added unit tests
NEW: InputDataAgent - possibility to refresh only data registered in the last predefined period of time 
NEW: TransformationAgent(Client) - management of derived transformations and more ported from LHCbDIRAC
BUGFIX: TransformationDB - wrong SQL statement generation in setFileStatusForTransformation()

[v6r5p14]

*Core
NEW: Utilities - added Backports utility

*WMS
FIX: Use /Operations/JobScheduling section consistently, drop /Operations/Matching section
NEW: Allow VO specific share correction plugins from extensions
FIX: Executors - several fixes

[v6r5p13]

*WMS
FIX: Executors - VOPlugin will properly send and receive the params
NEW: Correctors can be defined in an extension
FIX: Correctors - Properly retrieve info from the CS using the ops helper

[v6r5p12]

FIX: merged in patch v6r4p34

[v6r5p11]

FIX: merged in patch v6r4p33

*Core
FIX: MySQL - added offset argument to buildConditions()

[v6r5p10]

FIX: merged in patch v6r4p32

[v6r5p9]

FIX: merged in patch v6r4p30

[v6r5p8]

FIX: merged in patch v6r4p29

[v6r5p7]

FIX: merged in patch v6r4p28

[v6r5p6]

FIX: merged in patch v6r4p27

*Transformation
BUGFIX: TransformationDB - StringType must be imported before it can be used

*RSS
NEW: CS.py - Space Tokens were hardcoded, now are obtained after scanning the StorageElements.

[v6r5p5]

FIX: merged in patch v6r4p26

[v6r5p4]

FIX: merged in patch v6r4p25

[v6r5p3]

*Transformation
FIX: merged in patch v6r4p24

[v6r5p2]

*Web
NEW: includes DIRACWeb tag web2012092101

[v6r5p1]

*Core
BUGFIX: ExecutorMindHandler - return S_OK() in the initializeHandler
FIX: OptimizationMindHandler - if the manifest is not dirty it will not be updated by the Mind

*Configuration
NEW: Resources helper - added getCompatiblePlatform(), getDIRACPlatform() methods

*Resources
FIX: SSHComputingElement - add -q option to ssh command to avoid banners in the output
FIX: BOINCComputingElement - removed debugging printout
FIX: ComputingElement - use Platform CS option which will be converted to LHCbPlatform for legacy compatibility

*DMS
FIX: RequestAgentBase - lowering loglevel from ALWAYS to INFO to avoid flooding SystemLogging

*WMS:
FIX: SiteDirector - provide CE platform parameter when interrogating the TQ
FIX: GridPilotDirector - publish pilot OwnerGroup rather than VOMS role
FIX: WMSUtilities - add new error string into the parsing of the job output retrieval

[v6r5]

NEW: Executor framework

*Core
NEW: MySQL.py - added Test case for Time.dateTime time stamps
NEW: MySQL.py - insertFields and updateFields can get values via Lists or Dicts
NEW: DataIntegrityDB - use the new methods from MySQL and add test cases
NEW: DataIntegrityHandler - check connection to DB and create tables (or update their schema)
NEW: DataLoggingDB - use the new methods from MySQL and add test cases
NEW: DataLoggingHandler - check connection to DB and create tables (or update their schema)
FIX: ProcessPool - killing stuck workers after timeout
CHANGE: DB will throw a RuntimeException instead of a sys.exit in case it can't contact the DB
CHANGE: Several improvements on DISET
CHANGE: Fixed all DOS endings to UNIX
CHANGE: Agents, Services and Executors know how to react to CSSection/Module and react accordingly
NEW: install tools are updated to deal with executors
FIX: dirac-install - add -T/--Timeout option to define timeout for distribution downloads
NEW: dirac-install - added possibility of defining dirac-install's global defaults by command line switch
BUGFIX: avoid PathFinder.getServiceURL and use Client class ( DataLoggingClient,LfcFileCatalogProxyClient ) 
FIX: MySQL - added TIMESTAMPADD and TIMESTAMPDIFF to special values not to be scaped by MySQL
NEW: ObjectLoader utility
CHANGE: dirac-distribution - added global defaults flag and changed the flag to -M or --defaultsURL
FIX: Convert to string before trying to escape value in MySQL
NEW: DISET Services - added PacketTimeout option
NEW: SystemLoggingDB - updated to use the renewed MySQL interface and SQL schema
NEW: Added support for multiple entries in /Registry/DefaultGroup, for multi-VO installations
CHANGE: Component installation procedure updated to cope with components inheriting Modules
CHANGE: InstallTools - use dirac- command in runit run scripts
FIX: X509Chain - avoid a return of error when the group is not valid
FIX: MySQL - reduce verbosity of log messages when high level methods are used
CHANGE: Several DB classes have been updated to use the MySQL buildCondition method
NEW: MySQL - provide support for greater and smaller arguments to all MySQL high level methods
FIX: Service.py - check all return values from all initializers

*Configuration
CHANGE: By default return option and section lists ordered as in the CS
NEW: ConfigurationClient - added function to refresh remote configuration

*Framework
FIX: Registry.findDefaultGroup will never return False
CHANGE: ProxyManager does not accept proxies without explicit group
CHANGE: SystemAdministratorHandler - force refreshing the configuration after new component setup

*RSS
CHANGE: removed code execution from __init__
CHANGE: removed unused methods
NEW: Log all policy results 

*Resources
NEW: updated SSHComputingElement which allows multiple job submission
FIX: SGETimeLeft - better parsing of the batch system commands output
FIX: InProcessComputingElement - when starting a new job discard renewal of the previous proxy
NEW: BOINCComputingElement - new CE client to work with the BOINC desktop grid infrastructure 

*WMS
CHANGE: WMS Optimizers are now executors
CHANGE: SandboxStoreClient can directly access the DB if available
CHANGE: Moved JobDescription and improved into JobManifest
FIX: typo in JobLoggingDB
NEW: JobState/CachedJobState allow access to the Job via DB/JobStateSync Service automatically
BUGFIX: DownloadInputData - when not enough disk space, message was using "buffer" while it should be using "data"
FIX: the sandboxmetadataDB explosion when using the sandboxclient without direct access to the DB
NEW: Added support for reset/reschedule in the OptimizationMind
CHANGE: Whenever a DB is not properly initialized it will raise a catchable RuntimeError exception 
        instead of silently returning
FIX: InputDataResolution - just quick mod for easier extensibility, plus removed some LHCb specific stuff
NEW: allow jobids in a file in dirac-wms-job-get-output
NEW: JobManager - zfill in %n parameter substitution to allow alphabetical sorting
NEW: Directors - added checking of the TaskQueue limits when getting eligible queues
CHANGE: Natcher - refactor to simpify the logic, introduced Limiter class
CHANGE: Treat MaxCPUTime and CPUTime the same way in the JDL to avoid confusion
NEW: SiteDirector - added options PilotScript, MaxPilotsToSubmit, MaxJobsInFillMode
BUGFIX: StalledJobAgent - use cpuNormalization as float, not string 
FIX: Don't kill an executor if a task has been taken out from it
NEW: dirac-boinc-pilot - pilot script to be used on the BOINC volunteer nodes
FIX: SiteDirector - better handling of tokens and filling mode 
NEW: Generic pilot identities are automatically selected by the TQD and the SiteDirector 
     if not explicitly defined in /Pilot/GenericDN and GenericGroup
NEW: Generic pilot groups can have a VO that will be taken into account when selecting generic 
     credentials to submit pilots
NEW: Generic pilots that belong to a VO can only match jobs from that VO
NEW: StalledJobAgent - added rescheduling of jobs stuck in Matched or Rescheduled status
BUGFIX: StalledJobAgent - default startTime and endTime to "now", avoid None value
NEW: JobAgent - stop after N failed matching attempts (nothing to do), use StopAfterFailedMatches option
CHANGE: JobAgent - provide resource description as a dictionary to avoid extra JDL parsing by the Matcher
CHANGE: Matcher - report pilot info once instead of sending it several times from the job
CHANGE: Matcher - set the job site instead of making a separate call to JobStateUpdate
NEW: Matcher - added Matches done and matches OK statistics
NEW: TaskQueue - don't delete fresh task queues. Wait 5 minutes to do so.
CHANGE: Disabled TQs can also be matched, if no jobs are there, a retry will be triggered

*Transformation
FIX: TransformationAgent - a small improvement: now can pick the prods status to handle from the CS, 
     plus few minor corrections (e.g. logger messages)
FIX: TransformationCLI - take into accout possible failures in resetFile command     

*Accounting
NEW: AccountingDB - added retrieving RAW records for internal stuff
FIX: AccountingDB - fixed some logic for readonly cases
CHANGE: Added new simpler and faster bucket insertion mechanism
NEW: Added more info when rebucketing
FIX: Calculate the rebucket ETA using remaining records to be processed instead of the total records to be processed
FIX: Plots with no data still carry the plot name

*DMS
NEW: SRM2Storage - added retry in the gfal calls
NEW: added new FTSCleaningAgent cleaning up TransferDB tables
FIX: DataLoggingClient and DataLoggingDB - tests moved to separate files
CHANGE: request agents cleanup

*RMS
CHANGE: Stop using RequestAgentMixIn in the request agents

[v6r4p34]

*DMS
BUGFIX: FileCatalogCLI - fixed wrong indentation
CHANGE: RegistrationTask - removed some LHCb specific defaults

[v6r4p33]

*DMS
CHANGE: FTSRequest - be more verbose if something is wrong with file

[v6r4p32]

*WMS
FIX: StalledJobAgent - avoid exceptions in the stalled job accounting reporting

*DMS
NEW: FTSMonitorAgent - handling of expired FTS jobs 

*Interfaces
CHANGE: Dirac.py - attempt to retrieve output sandbox also for Completed jobs in retrieveRepositorySandboxes()

[v6r4p30]

*Core
BUGFIX: dirac-admin-bdii-ce-voview - proper check of the result structure

*Interfaces
FIX: Dirac.py, Job.py - allow to pass environment variables with special characters

*DMS
NEW: FileCatalogCLI - possibility to sort output in the ls command

*WMS:
FIX: JobWrapper - interpret environment variables with special characters 

[v6r4p29]

*RMS
BUGFIX: RequestDBMySQL - wrong indentation in __updateSubRequestFiles()

[v6r4p28]

*Interfaces
CHANGE: Dirac.py, DiracAdmin.py - remove explicit timeout on RPC client instantiation

*RSS
FIX: CS.py - fix for updated CS location (backward compatible)

*DMS
BUGFIX: StrategyHandler - bug fixed determineReplicationTree()
FIX: FTSRequest - add checksum string to SURLs file before submitting an FTS job

*WMS
FIX: JobWrapper - protection for double quotes in JobName
CHANGE: SiteDirector - switched some logging messages from verbose to info level

*RMS
NEW: Request(Client,DBMySQL,Manager) - added readRequestsForJobs() method

[v6r4p27]

*DMS
FIX: SRM2Storage - removed hack for EOS (fixed server-side)

*Transformation
CHANGE: TransformationClient - limit to 100 the number of transformations in getTransformations()
NEW: TransformationAgent - define the transformations type to use in the configuration

*Interfaces
FIX: Job.py -  fix for empty environmentDict (setExecutionEnv)

[v6r4p26]

*Transformation
BUGFIX: TransformationClient - fixed calling sequence in rpcClient.getTransformationTasks()
NEW: TransformationClient - added log messages in verbose level.

[v6r4p25]

*DMS
BUGFIX: StrategyHandler - sanity check for wrong replication tree 

[v6r4p24]

*Core
NEW: MySQL - add 'offset' argument to the buildCondition()

*Transformation
FIX: TransformationAgent - randomize the LFNs for removal/replication case when large number of those
CHANGE: TransformationClient(DB,Manager) - get transformation files in smaller chunks to
        improve performance
FIX: TransformationAgent(DB) - do not return redundant LFNs in getTransformationFiles()    

[v6r4p23]

*Web
NEW: includes DIRACWeb tag web2012092101

[v6r4p22]

*DMS
FIX: SRM2Storage - fix the problem with the CERN-EOS storage 

[v6r4p21]

*Core
BUGFIX: SGETimeLeft - take into account dd:hh:mm:ss format of the cpu consumed

[v6r4p20]

*WMS
BUGFIX: PilotDirector, GridPilotDirector - make sure that at least 1 pilot is to be submitted
BUGFIX: GridPilotDirector - bug on how pilots are counted when there is an error in the submit loop.
BUGFIX: dirac-pilot - proper install script installation on OSG sites

[v6r4p19]

*RMS
FIX: RequestDBMySQL - optimized request selection query 

[v6r4p18]

*Configuration
BUGFIX: CE2CSAgent.py - the default value must be set outside the loop

*DMS
NEW: dirac-dms-create-replication-request
BUGFIX: dirac-dms-fts-submit, dirac-dms-fts-monitor - print out error messages

*Resources
BUGFIX: TorqueComputingElement.py, plus add UserName for shared Queues

*WMS
BUGFIX: JobManagerHandler - default value for pStart (to avoid Exception)

[v6r4p17]

*Core
FIX: dirac-configure - setup was not updated in dirac.cfg even with -F option
FIX: RequestHandler - added fix for Missing ConnectionError

*DMS
FIX: dirac-dms-clean-directory - command fails with `KeyError: 'Replicas'`.

*WMS
FIX: SiteDirector - adapt to the new method in the Matcher getMatchingTaskQueue 
FIX: SiteDirector - added all SubmitPools to TQ requests

[v6r4p16]

*Core:
FIX: dirac-install - bashrc/cshrc were wrongly created when using versionsDir

*Accounting
CHANGE: Added new simpler and faster bucket insertion mechanism
NEW: Added more info when rebucketing

*WMS
CHANGE: Matcher - refactored to take into account job limits when providing info to directors
NEW: JoAgent - reports SubmitPool parameter if applicable
FIX: Matcher - bad codition if invalid result

[v6r4p15]

*WMS
FIX: gLitePilotDirector - fix the name of the MyProxy server to avoid crasehs of the gLite WMS

*Transformation
FIX: TaskManager - when the file is on many SEs, wrong results were generated

[v6r4p13]

*DMS
FIX: dirac-admin-allow-se - added missing interpreter line

[v6r4p12]

*DMS
CHANGE: RemovalTask - for DataManager shifter change creds after failure of removal with her/his proxy.

*RSS
NEW: Added RssConfiguration class
FIX: ResourceManagementClient  - Fixed wrong method name

[v6r4p11]

*Core
FIX: GGUSTicketsClient - GGUS SOAP URL updated

*DMS
BUGFIX: ReplicaManager - wrong for loop

*RequestManagement
BUGFIX: RequestClient - bug fix in finalizeRequest()

*Transformation
FIX: TaskManager - fix for correctly setting the sites (as list)

[v6r4p10]

*RequestManagement
BUGFIX: RequestContainer - in addSubrequest() function

*Resources
BUGFIX: SRM2Storage - in checksum type evaluation

*ResourceStatusSystem
BUGFIX: InfoGetter - wrong import statement

*WMS
BUGFIX: SandboxMetadataDB - __init__() can not return a value

[v6r4p9]

*DMS
CHANGE: FailoverTransfer - ensure the correct execution order of the subrequests

[v6r4p8]

Bring in fixes from v6r3p17

*Core:
FIX: Don't have the __init__ return True for all DBs
NEW: Added more protection for exceptions thrown in callbacks for the ProcessPool
FIX: Operations will now look in 'Defaults' instead of 'Default'

*DataManagement:
FIX: Put more protection in StrategyHandler for neither channels  not throughput read out of TransferDB
FIX: No JobIDs supplied in getRequestForJobs function for RequestDBMySQL taken into account
FIX: Fix on getRequestStatus
CHANGE: RequestClient proper use of getRequestStatus in finalizeRequest
CHANGE: Refactored RequestDBFile

[v6r4p7]

*WorkloadManagement
FIX: SandboxMetadataDB won't explode DIRAC when there's no access to the DB 
CHANGE: Whenever a DB fails to initialize it raises a catchable exception instead of just returning silently

*DataManagement
CHANGE: Added Lost and Unavailable to the file metadata

[v6r4p6]

Bring fixes from v6r4p6

[v6r4p5]

*Configuration
NEW: Added function to generate Operations CS paths

*Core
FIX: Added proper ProcessPool checks and finalisation

*DataManagement
FIX: don't set Files.Status to Failed for non-existign files, failover transfers won't go
FIX: remove classmethods here and there to unblock requestHolder
CHANGE: RAB, TA: change task timeout: 180 and 600 (was 600 and 900 respectively)
FIX: sorting replication tree by Ancestor, not hopAncestorgit add DataManagementSystem/Agent/TransferAgent.py
NEW: TA: add finalize
CHANGE: TransferAgent: add AcceptableFailedFiles to StrategyHandler to ban FTS channel from scheduling
FIX: if there is no failed files, put an empty dict


*RSS
FIX: RSS is setting Allowed but the StorageElement checks for Active

*Workflows
FIX: Part of WorfklowTask rewritten to fix some issues and allow 'ANY' as site

*Transformation
FIX: Wrong calls to TCA::cleanMetadataCatalogFiles

[v6r4p4]

*Core
FIX: Platform.py - check if Popen.terminate is available (only from 2.6)

[v6r4p3]

*Core
FIX: ProcessPool with watchdog and timeouts - applied in v6r3 first

[v6r4p2]

*StorageManagement
BUGFIX: StorageElement - staging is a Read operation and should be allowed as such

*WMS
BUGFIX: InProcessComputingElement, JobAgent - proper return status code from the job wrapper

*Core
FIX: Platform - manage properly the case of exception in the ldconfig execution

[v6r4p1]

*DMS
FIX: TransferDB.getChannelObservedThroughput - the channelDict was created in a wrong way

*RSS
FIX: ResourceStatus was not returning Allowed by default

[v6r4]

*Core
FIX: dirac-install-db.py: addDatabaseOptionsToCS has added a new keyed argument
NEW: SGETimeLeft.py: Support for SGE backend
FIX: If several extensions are installed, merge ConfigTemplate.cfg
NEW: Service framework - added monitoring of file descriptors open
NEW: Service framework - Reduced handshake timeout to prevent stuck threads
NEW: MySQL class with new high level methods - buildCondition,insertFields,updateFields
     deleteEntries, getFields, getCounters, getDistinctAttributeValues
FIX: ProcessPool - fixes in the locking mechanism with LockRing, stopping workers when the
     parent process is finished     
FIX: Added more locks to the LockRing
NEW: The installation tools are updated to install components by name with the components module specified as an option

*DMS
FIX: TransferDB.py - speed up the Throughput determination
NEW: dirac-dms-add-files: script similar to dirac-dms-remove-files, 
     allows for 1 file specification on the command line, using the usual dirac-dms-add-file options, 
     but also can take a text file in input to upload a bunch of files. Exit code is 0 only if all 
     was fine and is different for every error found. 
NEW: StorageElementProxy- support for data downloading with http protocol from arbitrary storage, 
     needed for the web data download
BUGFIX: FileCatalogCLI - replicate operation does a proper replica registration ( closes #5 )     
FIX: ReplicaManager - __cleanDirectory now working and thus dirac-dms-clean-directory

*WMS
NEW: CPU normalization script to run a quick test in the pilot, used by the JobWrapper
     to report the CPU consumption to the accounting
FIX: StalledJobAgent - StalledTimeHours and FailedTimeHours are read each cycle, refer to the 
     Watchdog heartBeat period (should be renamed); add NormCPUTime to Accounting record
NEW: SiteDirector - support for the operation per VO in multi-VO installations
FIX: StalledJobAgent - get ProcessingType from JDL if defined
BUGFIX: dirac-wms-job-peek - missing printout in the command
NEW: SiteDirector - take into account the number of already waiting pilots when evaluating the number of pilots to submit
FIX: properly report CPU usage when the Watchdog kill the payload.

*RSS
BUGFIX: Result in ClientCache table is a varchar, but the method was getting a datetime
NEW: CacheFeederAgent - VOBOX and SpaceTokenOccupancy commands added (ported from LHCbDIRAC)
CHANGE: RSS components get operational parameters from the Operations handler

*DataManagement
FIX: if there is no failed files, put an empty dict

*Transformation
FIX: Wrong calls to TCA::cleanMetadataCatalogFiles

[v6r3p19]

*WMS
FIX: gLitePilotDirector - fix the name of the MyProxy server to avoid crashes of the gLite WMS

[v6r3p18]

*Resources
BUGFIX: SRM2Storage - in checksum type evaluation

[v6r3p17]

*DataManagement
FIX: Fixes issues #783 and #781. Bugs in ReplicaManager removePhisicalReplica and getFilesFromDirectory
FIX: Return S_ERROR if missing jobid arguments
NEW: Checksum can be verified during FTS and SRM2Storage 

[v6r3p16]

*DataManagement
FIX: better monitoring of FTS channels 
FIX: Handle properly None value for channels and bandwidths

*Core
FIX: Properly calculate the release notes if there are newer releases in the release.notes file

[v6r3p15]

*DataManagement
FIX: if there is no failed files, put an empty dict

*Transformation
FIX: Wrong calls to TCA::cleanMetadataCatalogFiles


[v6r3p14]

* Core

BUGFIX: ProcessPool.py: clean processing and finalisation
BUGFIX: Pfn.py: don't check for 'FileName' in pfnDict

* DMS

NEW: dirac-dms-show-fts-status.py: script showing last hour history for FTS channels
NEW: TransferDBMonitoringHandler.py: new function exporting FST channel queues
BUGFIX: TransferAgent.py,RemovalAgent.py,RegistrationAgent.py - unlinking of temp proxy files, corection of values sent to gMonitor
BUGFIX: StrategyHandler - new config option 'AcceptableFailedFiles' to unblock scheduling for channels if problematic transfers occured for few files
NEW: TransferAgent,RemovalAgent,RegistrationAgent - new confing options for setting timeouts for tasks and ProcessPool finalisation
BUGFIX: ReplicaManager.py - reverse sort of LFNs when deleting files and directories to avoid blocks
NEW: moved StrategyHandler class def to separate file under DMS/private

* TMS

FIX: TransformationCleaningAgent.py: some refactoring, new way of disabling/enabline execution by 'EnableFlag' config option

[v6r3p13]

*Core
FIX: Added proper ProcessPool checks and finalisation

*DataManagement
FIX: don't set Files.Status to Failed for non-existign files, failover transfers won't go
FIX: remove classmethods here and there to unblock requestHolder
CHANGE: RAB, TA: change task timeout: 180 and 600 (was 600 and 900 respectively)
FIX: sorting replication tree by Ancestor, not hopAncestorgit add DataManagementSystem/Agent/TransferAgent.py
NEW: TA: add finalize
CHANGE: TransferAgent: add AcceptableFailedFiles to StrategyHandler to ban FTS channel from scheduling

[v6r3p12]

*Core
FIX: Platform.py - check if Popen.terminate is available (only from 2.6)

[v6r3p11]

*Core
FIX: ProcessPool with watchdog and timeouts

[v6r3p10]

*StorageManagement
BUGFIX: StorageElement - staging is a Read operation and should be allowed as such

*WMS
BUGFIX: InProcessComputingElement, JobAgent - proper return status code from the job wrapper

*Core
FIX: Platform - manage properly the case of exception in the ldconfig execution

[v6r3p9]

*DMS
FIX: TransferDB.getChannelObservedThroughput - the channelDict was created in a wrong way

[v6r3p8]

*Web
CHANGE: return back to the release web2012041601

[v6r3p7]

*Transformation
FIX: TransformationCleaningAgent - protection from deleting requests with jobID 0 

[v6r3p6]

*Core
FIX: dirac-install-db - proper key argument (follow change in InstallTools)
FIX: ProcessPool - release all locks every time WorkignProcess.run is executed, more fixes to come
FIX: dirac-configure - for Multi-Community installations, all vomsdir/vomses files are now created

*WMS
NEW: SiteDirector - add pilot option with CE name to allow matching of SAM jobs.
BUGFIX: dirac-pilot - SGE batch ID was overwriting the CREAM ID
FIX: PilotDirector - protect the CS master if there are at least 3 slaves
NEW: Watchdog - set LocalJobID in the SGE case

[v6r3p5]

*Core:
BUGFIX: ProcessPool - bug making TaskAgents hang after max cycles
BUGFIX: Graphs - proper handling plots with data containing empty string labels
FIX: GateWay - transfers were using an old API
FIX: GateWay - properly calculate the gateway URL
BUGFIX: Utilities/Pfn.py - bug in pfnunparse() when concatenating Path and FileName

*Accounting
NEW: ReportGenerator - make AccountingDB readonly
FIX: DataCache - set daemon the datacache thread
BUGFIX: BasePlotter - proper handling of the Petabyte scale data

*DMS:
BUGFIX: TransferAgent, RegistrationTask - typos 

[v6r3p4]

*DMS:
BUGFIX: TransferAgent - wrong value for failback in TA:execute

[v6r3p3]

*Configuration
BUGFIX: Operations helper - typo

*DMS:
FIX: TransferAgent - change the way of redirecting request to task

[v6r3p2]

*DMS
FIX: FTSRequest - updating metadata for accouting when finalizing FTS requests

*Core
FIX: DIRAC/__init__.py - default version is set to v6r3

[v6r3p1]

*WMS
CHANGE: Use ResourcesStatus and Resources helpers in the InputDataAgent logic

*Configuration
NEW: added getStorageElementOptions in Resources helper

*DMS
FIX: resourceStatus object created in TransferAgent instead of StrategyHandler

[v6r3]

*Core
NEW: Added protections due to the process pool usage in the locking logic

*Resources
FIX: LcgFileCatalogClient - reduce the number of retries: LFC_CONRETRY = 5 to 
     avoid combined catalog to be stuck on a faulty LFC server
     
*RSS
BUGFIX: ResourceStatus - reworked helper to keep DB connections     

*DMS
BUGFIX: ReplicaManager::CatalogBase::_callFileCatalogFcnSingleFile() - wrong argument

*RequestManagement
FIX: TaskAgents - set timeOut for task to 10 min (15 min)
NEW: TaskAgents - fill in Error fields in case of failing operations

*Interfaces
BUGFIX: dirac-wms-select-jobs - wrong use of the Dirac API

[v6r2p9]

*Core
FIX: dirac-configure - make use of getSEsForSite() method to determine LocalSEs

*WMS
NEW: DownloadInputData,InputDataByProtocol - check Files on Tape SEs are on Disk cache 
     before Download or getturl calls from Wrapper
CHANGE: Matcher - add Stalled to "Running" Jobs when JobLimits are applied   
CHANGE: JobDB - allow to specify required platform as Platform JDL parameter,
        the specified platform is taken into account even without /Resources/Computing/OSCompatibility section

*DMS
CHANGE: dirac-admin-allow(ban)-se - removed lhcb-grid email account by default, 
        and added switch to avoid sending email
FIX: TaskAgents - fix for non-existing files
FIX: change verbosity in failoverReplication 
FIX: FileCatalog - remove properly metadata indices 
BUGFIX: FileManagerBase - bugfix in the descendants evaluation logic  
FIX: TransferAgent and TransferTask - update Files.Status to Failed when ReplicaManager.replicateAndRegister 
     will fail completely; when no replica is available at all.

*Core
FIX: dirac-pilot - default lcg bindings version set to 2012-02-20

[v6r2p8]

*DMS:
CHANGE: TransferAgent - fallback to task execution if replication tree is not found

[v6r2p7]

*WMS
BUGFIX: SiteDirector - wrong CS option use: BundleProxy -> HttpProxy
FIX: SiteDirector - use short lines in compressed/encoded files in the executable
     python script

[v6r2p6]

*DataManagement
FIX: Bad logic in StrategyHandler:MinimiseTotalWait

*Core
CHANGE: updated GGUS web portal URL

*RSS
BUGFIX: meta key cannot be reused, it is popped from dictionary

*Framework
FIX: The Gateway service does not have a handler
NEW: ConfingTemplate entry for Gateway
FIX: distribution notes allow for word wrap

*WorkloadManagement
FIX: avoid unnecessary call if no LFN is left in one of the SEs
FIX: When Uploading job outputs, try first Local SEs, if any


[v6r2p5]

*RSS
BUGFIX: several minor bug fixes

*RequestManagement
BUGFIX: RequestDBMySQL - removed unnecessary request type check

*DMS
BUGFIX: FileCatalogClienctCLI - wrong evaluation of the operation in the find command
NEW: FileCatalog - added possibility to remove specified metadata for a given path 
BUGFIX: ReplicaManager - wrong operation order causing failure of UploadLogFile module

*Core
NEW: dirac-install - generate cshrc DIRAC environment setting file for the (t)csh 

*Interfaces
CHANGE: Job - added InputData to each element in the ParametricInputData

*WMS
CHANGE: dirac-jobexec - pass ParametericInputData to the workflow as a semicolon separated string

[v6r2p4]

*WMS
BUGFIX: StalledJobAgent - protection against jobs with no PilotReference in their parameters
BUGFIX: WMSAdministratorHandler - wrong argument type specification for getPilotInfo method

*StorageManagement
BUGFIX: RequestFinalizationAgent - no method existence check when calling RPC method

[v6r2p3]

*WMS
CHANGE: Matcher - fixed the credentials check in requestJob() to simplify it

*ConfigurationSystem
CHANGE: Operations helper - fix that allow no VO to be defined for components that do not need it

*Core
BUGFIX: InstallTools - when applying runsvctrl to a list of components make sure that the config server is treated first and the sysadmin service - last
        
[v6r2p2]

*WMS
BUGFIX: Matcher - restored logic for checking private pilot asking for a given DN for belonging to the same group with JOB_SHARING property.

[v6r2p1]

*RequestManagementSystem
BUGFIX: RequestCleaningAgent - missing import of the "second" interval definition 

[v6r2]

*General
FIX: replaced use of exec() python statement in favor of object method execution

*Accounting
CHANGE: Accounting 'byte' units are in powers of 1000 instead of powers of 1024 (closes #457)

*Core
CHANGE: Pfn.py - pfnparse function rewritten for speed up and mem usage, unit test case added
FIX: DISET Clients are now thread-safe. Same clients used twice in different threads was not 
closing the previous connection
NEW: reduce wait times in DISET protocol machinery to improve performance    
NEW: dirac-fix-mysql-script command to fix the mysql start-up script for the given installation
FIX: TransferClient closes connections properly
FIX: DISET Clients are now thread-safe. Same client used twice in different threads will not close the previous connection
CHANGE: Beautification and reduce wait times to improve performance
NEW: ProcessPool - added functionality to kill all children processes properly when destroying ProcessPool objects
NEW: CS Helper for LocalSite section, with gridEnv method
NEW: Grid module will use Local.gridEnv if nothing passed in the arguments
CHANGE: Add deprecated sections in the CS Operations helper to ease the transition
FIX: dirac-install - execute dirac-fix-mysql-script, if available, to fix the mysql.server startup script
FIX: dirac-distribution - Changed obsoleted tar.list file URL
FIX: typo in dirac-admin-add-host in case of error
CHANGE: dirac-admin-allow(ban)-se - use diracAdmin.sendMail() instead of NotificationClient.sendMail()

*Framework
BUGFIX: UserProfileDB - no more use of "type" variable as it is a reserved keyword 

*RequestManagement:
FIX: RequestDBFile - more consistent treatment of requestDB Path
FIX: RequestMySQL - Execution order is evaluated based on not Done state of subrequests
NEW: RequestCleaningAgent - resetting Assigned requests to Waiting after a configurable period of time

*RSS
CHANGE: RSS Action now inherits from a base class, and Actions are more homogeneous, they all take a uniform set of arguments. The name of modules has been changed from PolType to Action as well.
FIX: CacheFeederAgent - too verbose messages moved to debug instead of info level
BUGFIX: fixed a bug preventing RSS clients to connect to the services     
FIX: Proper services synchronization
FIX: Better handling of exceptions due to timeouts in GOCDBClient   
FIX: RSS.Notification emails are sent again
FIX: Commands have been modified to return S_OK, S_ERROR inside the Result dict. This way, policies get a S_ERROR / S_OK object. CacheFeederAgent has been updated accordingly.
FIX: allow clients, if db connection fails, to reconnect ( or at least try ) to the servers.
CHANGE: access control using CS Authentication options. Default is SiteManager, and get methods are all.
BUGFIX: MySQLMonkey - properly escaped all parameters of the SQL queries, other fixes.
NEW: CleanerAgent renamed to CacheCleanerAgent
NEW: Updated RSS scripts, to set element statuses and / or tokens.
NEW: Added a new script, dirac-rss-synch
BUGFIX: Minor bugfixes spotted on the Web development
FIX: Removed useless decorator from RSS handlers
CHANGE: ResourceStatus helper tool moved to RSS/Client directory, no RSS objects created if the system is InActive
CHANGE: Removed ClientFastDec decorator, using a more verbose alternative.
CHANGE: Removed useless usage of kwargs on helper functions.  
NEW: added getSESitesList method to RSSClient      
FIX: _checkFloat() checks INTEGERS, not datetimes

*DataManagement
CHANGE: refactoring of DMS agents executing requests, allow requests from arbitrary users
NEW: DFC - allow to specify multiple replicas, owner, mode when adding files
CHANGE: DFC - optimization of the directory size evaluation
NEW: Added CREATE TEMPORARY TABLES privilege to FileCatalogDB
CHANGE: DFC - getCatalogCounters() update to show numbers of directories
NEW: lfc_dfc_copy script to migrate data from LFC to DFC
FIX: dirac-dms-user-lfns - fixed the case when the baseDir is specified
FIX: FTS testing scripts were using sys.argv and getting confused if options are passed
NEW: DFC - use DirectoryUsage tables for the storage usage evaluations
NEW: DFC - search by metadata can be limited to a given directory subtree
NEW: DFC - search by both directory and file indexed metadata
BUGFIX: DFC - avoid crash if no directories or files found in metadata query
NEW: DFC FileCatalogHandler - define database location in the configuration
NEW: DFC - new FileCatalogFactory class, possibility to use named DFC services
FIX: FTSMonitor, FTSRequest - fixes in handling replica registration, setting registration requests in FileToCat table for later retry
FIX: Failover registration request in the FTS agents.      
FIX: FTSMonitor - enabled to register new replicas if even the corresponding request were removed from the RequestManagement 
FIX: StorageElement - check if SE has been properly initialized before executing any method     
CHANGE: LFC client getReplica() - make use of the new bulk method lfc.lfc_getreplicasl()
FIX: LFC client - protect against getting None in lfc.lfc_readdirxr( oDirectory, "" )  
FIX: add extra protection in dump method of StorageElement base class
CHANGE: FailoverTransfer - create subrequest per catalog if more than one catalog

*Interface
NEW: Job.py - added method to handle the parametric parameters in the workflow. They are made available to the workflow_commons via the key 'GenericParameters'.
FIX: Dirac.py - fix some type checking things
FIX: Dirac.py - the addFile() method can now register to more than 1 catalog.

*WMS
FIX: removed dependency of the JobSchedulingAgent on RSS. Move the getSiteTier functionality to a new CS Helper.
FIX: WMSAdministratorHandler - Replace StringType by StringTypes in the export methods argument type
FIX: JobAgent - Set explicitly UseServerCertificate to "no" for the job executable
NEW: dirac-pilot - change directory to $OSG_WN_TMP on OSG sites
FIX: SiteDirector passes jobExecDir to pilot, this defaults to "." for CREAM CEs. It can be set in the CS. It will not make use of $TMPDIR in this case.
FIX: Set proper project and release version to the SiteDirector     
NEW: Added "JobDelay" option for the matching, refactored and added CS options to the matcher
FIX: Added installation as an option to the pilots and random MyProxyServer
NEW: Support for parametric jobs with parameters that can be of List type

*Resources
NEW: Added SSH Grid Engine Computing Element
NEW: Added SSH Computing Element
FIX: make sure lfc client will not try to connect for several days

*Transformation
FIX: TransformationDB - in setFileStatusForTransformation() reset ErrorCount to zero if "force" flag and    the new status is "unused"
NEW: TransformationDB - added support for dictionary in metadata for the InputDataQuery mechanism     

[v6r1p13]

*WMS
FIX: JobSchedulingAgent - backported from v6r2 use of Resources helper

[v6r1p12]

*Accounting
FIX: Properly delete cached plots

*Core
FIX: dirac-install - run externals post install after generating the versions dir

[v6r1p11]

*Core
NEW: dirac-install - caches locally the externals and the grid bundle
FIX: dirac-distribution - properly generate releasehistory and releasenotes

[v6r1p10]

*WorloadManagement
FIX: JobAgent - set UseServerCertificate option "no" for the job executable

[v6r1p9]

*Core
FIX: dirac-configure - set the proper /DIRAC/Hostname when defining /LocalInstallation/Host

*DataManagement
FIX: dirac-dms-user-lfns - fixed the case when the baseDir is specified
BUGFIX: dirac-dms-remove-files - fixed crash in case of returned error report in a form of dictionary 

[v6r1p8]

*Web
FIX: restored Run panel in the production monitor

*Resources
FIX: FileCatalog - do not check existence of the catalog client module file

[v6r1p7]

*Web
BUGFIX: fixed scroll bar in the Monitoring plots view

[v6r1p6]

*Core
FIX: TransferClient closes connections properly

[v6r1p5]

*Core
FIX: DISET Clients are now thread-safe. Same clients used twice in different threads was not 
     closing the previous connection
NEW: reduce wait times in DISET protocol machinery to improve performance   

[v6r1p4]

*RequestManagement
BUGFIX: RequestContainer - in isSubRequestDone() treat special case for subrequests with files

*Transformation
BUGFIX: TransformationCleaningAgent - do not clear requests for tasks with no associated jobs

[v6r1p3]

*Framework
NEW: Pass the monitor down to the request RequestHandler
FIX: Define the service location for the monitor
FIX: Close some connections that DISET was leaving open

[v6r1p2]

*WorkloadManagement
BUGFIX: JobSchedulingAgent - use getSiteTiers() with returned direct value and not S_OK

*Transformation
BUGFIX: Uniform use of the TaskManager in the RequestTaskAgent and WorkflowTaskAgent

[v6r1p1]

*RSS
BUGFIX: Alarm_PolType now really send mails instead of crashing silently.

[v6r1]

*RSS
CHANGE: Major refactoring of the RSS system
CHANGE: DB.ResourceStatusDB has been refactored, making it a simple wrapper round ResourceStatusDB.sql with only four methods by table ( insert, update, get & delete )
CHANGE: DB.ResourceStatusDB.sql has been modified to support different statuses per granularity.
CHANGE: DB.ResourceManagementDB has been refactored, making it a simple wrapper round ResourceStatusDB.sql with only four methods by table ( insert, update, get & delete )
CHANGE: Service.ResourceStatusHandler has been refactored, removing all data processing, making it an intermediary between client and DB.
CHANGE: Service.ResourceManagementHandler has been refactored, removing all data processing, making it an intermediary between client and DB.
NEW: Utilities.ResourceStatusBooster makes use of the 'DB primitives' exposed on the client and does some useful data processing, exposing the new functions on the client.
NEW: Utilities.ResourceManagementBooster makes use of the 'DB primitives' exposed on the client and does some useful data processing, exposing the new functions on the client.
CHANGE: Client.ResourceStatusClient has been refactorerd. It connects automatically to DB or to the Service. Exposes DB and booster functions.
CHANGE: Client.ResourceManagementClient has been refactorerd. It connects automatically to DB or to the Service. Exposes DB and booster functions.
CHANGE: Agent.ClientsCacheFeederAgent renamed to CacheFeederAgent. The name was not accurate, as it also feeds Accouting Cache tables.
CHANGE: Agent.InspectorAgent, makes use of automatic API initialization.
CHANGE: Command. refactor and usage of automatic API initialization.
CHANGE: PolicySystem.PEP has reusable client connections, which increase significantly performance.
CHANGE: PolicySystem.PDP has reusable client connections, which increase significantly performance.
NEW: Utilities.Decorators are syntactic sugar for DB, Handler and Clients.
NEW: Utilities.MySQLMonkey is a mixture of laziness and refactoring, in order to generate the SQL statements automatically. Not anymore sqlStatemens hardcoded on the RSS.
NEW: Utilities.Validator are common checks done through RSS modules
CHANGE: Utilities.Synchronizer syncs users and DIRAC sites
CHANGE: cosmetic changes everywhere, added HeadURL and RCSID
CHANGE: Removed all the VOExtension logic on RSS
BUGFIX: ResourceStatusHandler - getStorageElementStatusWeb(), access mode by default is Read
FIX: RSS __init__.py will not crash anymore if no CS info provided
BUGFIX: CS.getSiteTier now behaves correctly when a site is passed as a string

*dirac-setup-site
BUGFIX: fixed typos in the Script class name

*Transformation
FIX: Missing logger in the TaskManager Client (was using agent's one)
NEW: Added UnitTest class for TaskManager Client

*DIRAC API
BUGFIX: Dirac.py. If /LocalSite/FileCatalog is not define the default Catalog was not properly set.
FIX: Dirac.py - fixed __printOutput to properly interpret the first argument: 0:stdout, 1:stderr
NEW: Dirac.py - added getConfigurationValue() method

*Framework
NEW: UsersAndGroups agent to synchronize users from VOMRS server.

*dirac-install
FIX: make Platform.py able to run with python2.3 to be used inside dirac-install
FIX: protection against the old or pro links pointing to non-existent directories
NEW: make use of the HTTP proxies if available
FIX: fixed the logic of creating links to /opt/dirac directories to take into account webRoot subdirs

*WorkloadManagement
FIX: SiteDirector - change getVO() function call to getVOForGroup()

*Core:
FIX: Pfn.py - check the sanity of the pfn and catch the erroneous case

*RequestManagement:
BUGFIX: RequestContainer.isSubrequestDone() - return 0 if Done check fails

*DataManagement
NEW: FileCatalog - possibility to configure multiple FileCatalog services of the same type

[v6r0p4]

*Framework
NEW: Pass the monitor down to the request RequestHandler
FIX: Define the service location for the monitor
FIX: Close some connections that DISET was leaving open

[v6r0p3]

*Framework
FIX: ProxyManager - Registry.groupHasProperties() wasn't returning a result 
CHANGE: Groups without AutoUploadProxy won't receive expiration notifications 
FIX: typo dirac-proxy-info -> dirac-proxy-init in the expiration mail contents
CHANGE: DISET - directly close the connection after a failed handshake

[v6r0p2]

*Framework
FIX: in services logs change ALWAYS log level for query messages to NOTICE

[v6r0p1]

*Core
BUGFIX: List.uniqueElements() preserves the other of the remaining elements

*Framework
CHANGE: By default set authorization rules to authenticated instead of all
FIX: Use all required arguments in read access data for UserProfileDB
FIX: NotificationClient - dropped LHCb-Production setup by default in the __getRPSClient()

[v6r0]

*Framework
NEW: DISET Framework modified client/server protocol, messaging mechanism to be used for optimizers
NEW: move functions in DIRAC.Core.Security.Misc to DIRAC.Core.Security.ProxyInfo
CHANGE: By default log level for agents and services is INFO
CHANGE: Disable the log headers by default before initializing
NEW: dirac-proxy-init modification according to issue #29: 
     -U flag will upload a long lived proxy to the ProxyManager
     If /Registry/DefaultGroup is defined, try to generate a proxy that has that group
     Replaced params.debugMessage by gLogger.verbose. Closes #65
     If AutoUploadProxy = true in the CS, the proxy will automatically be uploaded
CHANGE: Proxy upload by default is one month with dirac-proxy-upload
NEW: Added upload of pilot proxies automatically
NEW: Print info after creating a proxy
NEW: Added setting VOMS extensions automatically
NEW: dirac-proxy-info can also print the information of the uploaded proxies
NEW: dirac-proxy-init will check that the lifetime of the certificate is less than one month and advise to renew it
NEW: dirac-proxy-init will check that the certificate has at least one month of validity
FIX: Never use the host certificate if there is one for dirac-proxy-init
NEW: Proxy manager will send notifications when the uploaded proxies are about to expire (configurable via CS)
NEW: Now the proxyDB also has a knowledge of user names. Queries can use the user name as a query key
FIX: ProxyManager - calculate properly the dates for credentials about to expire
CHANGE: ProxyManager will autoexpire old proxies, also auto purge logs
CHANGE: Rename dirac-proxy-upload to dirac-admin-proxy-upload
NEW: dirac-proxy-init will complain if the user certificate has less than 30 days
CHANGE: SecurityLogging - security log level to verbose
NEW: OracleDB - added Array type 
NEW: MySQL - allow definition of the port number in the configuration
FIX: Utilities/Security - hash VOMS Attributes as string
FIX: Utilities/Security - Generate a chain hash to discover if two chains are equal
NEW: Use chain has to discover if it has already been dumped
FIX: SystemAdministrator - Do not set  a default lcg version
NEW: SystemAdministrator - added Project support for the sysadmin
CHANGE: SysAdmin CLI - will try to connect to the service when setting the host
NEW: SysAdmin CLI - colorization of errors in the cli
NEW: Logger - added showing the thread id in the logger if enabled
     
*Configuration
NEW: added getVOfromProxyGroup() utility
NEW: added getVoForGroup() utility, use it in the code as appropriate
NEW: added Registry and Operations Configuration helpers
NEW: dirac-configuration-shell - a configuration script for CS that behaves like an UNIX shellCHANGE: CSAPI - added more functionality required by updated configuration console
NEW: Added possibility to define LocalSE to any Site using the SiteLocalSEMapping 
     section on the Operations Section     
NEW: introduce Registry/VO section, associate groups to VOs, define SubmitPools per VO
FIX: CE2CSAgent - update the CEType only if there is a relevant info in the BDII  

*ReleaseManagement
NEW: release preparations and installation tools based on installation packages
NEW: dirac-compile-externals will try go get a DIRAC-free environment before compiling
NEW: dirac-disctribution - upload command can be defined via defaults file
NEW: dirac-disctribution - try to find if the version name is a branch or a tag in git and act accordingly
NEW: dirac-disctribution - added keyword substitution when creating a a distribution from git
FIX: Install tools won't write HostDN to the configuration if the Admin username is not set 
FIX: Properly set /DIRAC/Configuration/Servers when installing a CS Master
FIX: install_site.sh - missing option in wget for https download: --no-check-certificate
FIX: dirac-install-agent(service) - If the component being installed already has corresponding 
     CS section, it is not overwritten unless explicitly asked for
NEW: dirac-install functionality enhancement: start using the switches as defined in issue #26;
CHANGE: dirac-install - write the defaults if any under defaults-.cfg so dirac-configure can 
        pick it up
FIX: dirac-install - define DYLD_LIBRARY_PATH ( for Mac installations )     
NEW: dirac-install - put all the goodness under a function so scripts like lhcb-proxy-init can use it easily
FIX: dirac-install - Properly search for the LcgVer
NEW: dirac-install will write down the releases files in -d mode   
CHANGE: use new dirac_install from gothub/integration branch in install_site.sh
NEW: Extensions can request custom external dependencies to be installed via pip when 
     installing DIRAC.
NEW: LCG bundle version can be defined on a per release basis in the releases.cfg 
NEW: dirac-deploy-scripts - when setting the lib path in the deploy scripts. 
     Also search for subpaths of the libdir and include them
NEW: Install tools - plainly separate projects from installations

*Accounting
CHANGE: For the WMSHistory type, send as JobSplitType the JobType
CHANGE: Reduced the size of the max key length to workaround mysql max bytes for index problem
FIX: Modified buckets width of 1week to 1 week + 1 day to fix summer time end week (1 hour more )

*WorkloadManagement
CHANGE: SiteDirector - simplified executable generation
NEW: SiteDirector - few more checks of error conditions   
NEW: SiteDirector - limit the queue max length to the value of MaxQueueLengthOption 
     ( 3 days be default )
BUGFIX: SiteDirector - do not download pilot output if the flag getPilotOutput is not set     
NEW: JobDB will extract the VO when applying DIRAC/VOPolicy from the proper VO
FIX: SSHTorque - retrieve job status by chunks of 100 jobs to avoid too long
NEW: glexecComputingElement - allow glexecComputingElement to "Reschedule" jobs if the Test of
     the glexec fails, instead of defaulting to InProcess. Controlled by
     RescheduleOnError Option of the glexecComputingElement
NEW: SandboxStore - create a different SBPath with the group included     
FIX: JobDB - properly treat Site parameter in the job JDL while rescheduling jobs
NEW: JobSchedulingAgent - set the job Site attribute to the name of a group of sites corresponding 
     to a SE chosen by the data staging procedure 
CHANGE: TimeLeft - call batch system commands with the ( default ) timeout 120 sec
CHANGE: PBSTimeLeft - uses default CPU/WallClock if not present in the output  
FIX: PBSTimeLeft - proper handling of (p)cput parameter in the batch system output, recovery of the
     incomplete batch system output      
NEW: automatically add SubmitPools JDL option of the job owner's VO defines it     
NEW: JobManager - add MaxParametericJobs option to the service configuration
NEW: PilotDirector - each SubmitPool or Middleware can define TargetGrids
NEW: JobAgent - new StopOnApplicationFailure option to make the agent exiting the loop on application failure
NEW: PilotAgentsDB - on demand retrieval of the CREAM pilot output
NEW: Pilot - proper job ID evaluation for the OSG sites
FIX: ComputingElement - fixed proxy renewal logic for generic and private pilots
NEW: JDL - added %j placeholder in the JDL to be replaced by the JobID
BUGFIX: DownloadInputData - bug fixed in the naming of downloaded files
FIX: Matcher - set the group and DN when a request gets to the matcher if the request is not 
     coming from a pilot
FIX: Matcher = take into account JobSharing when checking the owner for the request
CHANGE: PilotDirector, dirac-pilot - interpret -V flag of the pilot as Installation name

*DataManagement
FIX: FileCatalog/DiractoryLevelTree - consistent application of the max directory level using global 
     MAX_LEVELS variable
FIX: FileCatalog - Directory metadata is deleted together with the directory deletion, issue #40    
CHANGE: FileCatalog - the logic of the files query by metadata revisited to increase efficiency 
FIX: LcgFileCatalog - use lfcthr and call lfcthr.init() to allow multithread
     try the import only once and just when LcgFileCatalogClient class is intantiated
NEW: LcgFileCatalogClient - new version of getPathPermissions relying on the lfc_access method to solve the problem
     of multiple user DNs in LFC.     
FIX: StorageElement - get service CS options with getCSOption() method ( closes #97 )
FIX: retrieve FileCatalogs as ordered list, to have a proper default.
CHANGE: FileCatalog - allow up to 15 levels of directories
BUGFIX: FileCatalog - bug fixes in the directory removal methods (closes #98)
BUGFIX: RemovalAgent - TypeError when getting JobID in RemovalAgent
BUGFIX: RemovalAgent - put a limit to be sure the execute method will end after a certain number of iterations
FIX: DownloadInputData - when files have been uploaded with lcg_util, the PFN filename
     might not match the LFN file name
FIX: putting FTSMonitor web page back
NEW: The default file catalog is now determined using /LocalSite/FileCatalog. The old behavior 
     is provided as a fallback solution
NEW: ReplicaManager - can now deal with multiple catalogs. Makes sure the surl used for removal is 
the same as the one used for registration.   
NEW: PoolXMLCatalog - added getTypeByPfn() function to get the type of the given PFN  
NEW: dirac-dms-ban(allow)-se - added possibility to use CheckAccess property of the SE

*StorageManagement
FIX: Stager - updateJobFromStager(): only return S_ERROR if the Status sent is not
recognized or if a state update fails. If the jobs has been removed or
has moved forward to another status, the Stager will get an S_OK and
should forget about the job.
NEW: new option in the StorageElement configuration "CheckAccess"
FIX: Requests older than 1 day, which haven't been staged are retried. Tasks older than "daysOld" 
     number of days are set to Failed. These tasks have already been retried "daysOld" times for staging.
FIX: CacheReplicas and StageRequests records are kept until the pin has expired. This way the 
     StageRequest agent will have proper accounting of the amount of staged data in cache.
NEW: FTSCleaningAgent will allow to fix transient errors in RequestDB. At the moment it's 
     only fixing Requests for which SourceTURL is equal to TargetSURL.
NEW: Stager - added new command dirac-stager-stage-files          
FIX: Update Stager code in v6 to the same point as v5r13p37
FIX: StorageManager - avoid race condition by ensuring that Links=0 in the query while removing replicas

*RequestManagement
FIX: RequestDBFile - get request in chronological order (closes issue #84)
BUGFIX: RequestDBFile - make getRequest return value for getRequest the same as for

*ResourceStatusSystem
NEW: Major code refacoring. First refactoring of RSS's PEP. Actions are now function 
     defined in modules residing in directory "Actions".
NEW: methods to store cached environment on a DB and ge them.
CHANGE: command caller looks on the extension for commands.
CHANGE: RSS use now the CS instead of getting info from Python modules.
BUGFIX: Cleaned RSS scripts, they are still prototypes
CHANGE: PEP actions now reside in separate modules outside PEP module.
NEW: RSS CS module add facilities to extract info from CS.
CHANGE: Updating various RSS tests to make them compatible with
changes in the system.
NEW: CS is used instead of ad-hoc configuration module in most places.
NEW: Adding various helper functions in RSS Utils module. These are
functions used by RSS developers, including mainly myself, and are
totally independant from the rest of DIRAC.
CHANGE: Mostly trivial changes, typos, etc in various files in RSS     
CHANGE: TokenAgent sends e-mails with current status   

*Transformation
CHANGE: allow Target SE specification for jobs, Site parameter is not set in this case
CHANGE: TransformationAgent  - add new file statuses in production monitoring display
CHANGE: TransformationAgent - limit the number of files to be treated in TransformationAgent 
        for replication and removal (default 5000)
BUGFIX: TransformationDB - not removing task when site is not set
BUGFIX: TransformationCleaningAgent - archiving instead of cleaning Removal and Replication 
        transformations 
FIX: TransformationCleaningAgent - kill jobs before deleting them        

*Workflow
NEW: allow modules to define Input and Output parameters that can be
     used instead of the step_commons/workflow_commons (Workflow.py, Step.py, Module.py)

*Various fixes
BUGFIX: Mail.py uses SMTP class rather than inheriting it
FIX: Platform utility will properly discover libc version even for the new Ubuntu
FIX: Removed old sandbox and other obsoleted components