# Backup of DC meeting minutes

---------
## DC meeting 22 Jun 2022

## Core computing report

* 2022ab ended earlier than anticipated due to electricity costs.
* Last physics run processed. Debug and cosmic runs to come.

* Issue with file validation replicated to DESY-TAPE-SE
* Investigation on going. May be related to update on the certificate of the host.
* Q: Is the issue related to access with SRM?
* A (Ueda-san): Not SRM since Rucio can access, but related with our old gfal2 version (gfal2 2.19.0).

## Data production summary

* Concerns from physics about the projected timeline on MC15rd. How can we be faster?
* We need input from Miyake-san if feasible.
* Q: Resources are mostly full. Where would be the bottleneck?

* Longzhou from DP submitted bucket 34 hRAW skims today.

## Operation matters

### Expert shift report

**Expert of the week: Hirata-san**

* Empty slots for DP shift in the next two weeks. Expert shift covered.

* ProdID 25686-25687 failed due to Seg fault (BIIDCO-4303)
* Most jobs submitted to SIGNET. The error is similar to memory issue at the site (BIIDCO-4303).
* Q: Can we check if the disk space is the cause of the issue?
* A (Ono-san): We have confirmed with Marko that the space avaiable is enough, but the failure rate is high.

* Low transfer efficiency to multiple SEs due to downtime of KIT SEs.
* Ueda-san banned CEs at LCG.KIT-TARDIS.de manually since they were not treated for the downtime.
* Downtime is finished.

* Transfer failure in NTU-DATA/TMP-SE as source.
* Files are very old (`/belle/MC/release-01-00-00/
DB00000294/...`). Why?
* A (Cedric): Moving files for decomissioning of the SE. Files will be treated, no action from expert shifter.

* GGUS tickets: 1 ticket submitted and verified, 3 tickets open with no updates.

* 3 messages to the comp-users-forum this week.
* gb2_job_reschedule error. Ono-san reported JobManager was stuck from Jun 20-21.
* User jobs stuck in "Matched" after restart.
* A (Ono-san): Job optimizer works.
* A (Ueda-san): Job matched means the job is assigned, but the Pilot cannot retrieve the job. We should find the Pilot ID for looking at the log.

* Feedback to Rucio Accounting plots: Desire a function to see occupancy for each SE at a glance.
* A: Cedric will fix how the information is displayed.

* Alert: Frascati-TMP-SE SE is getting full.
* Banned for write.

### Site configuration

* Davs enabled as write protocol for CNAF-TMP-SE, IN2P3CC-TMP-SE and KEK-DISK-TMP-SE (including third-party-copy among them).
* Performance is good between them after one week.
* Next candidate is UVic.
* Q: Can we enable Davs for TPC in other places?
* A (Ueda-san): Not in sites with red in the matrix. Enabling many sites at the same time can mess up with the sync.
*

* LMU-LOCAL-SE is ready be configured in production. The involved institutions are LRZ-LMU. Would be ok call it LRZ-LMU-LOCAL-SE?

* Update of pledged line after Site Report 2022 is ongoing.
* Shares in Operation->Production->SEshares have been updated. Need to verify other shares.

### Other operation matters

* Follow-up in the load on AMGA: production is on-going. Large number of files from a single user (no longer in Belle II) must be removed.

* Follow-up of v5r4p2
* Lost files during (after) production due to feature in gbasf2 that remove existing file.
* Fixed by PR#607 and deployed in v5r4p2 at May 31st.
* Running test script to verify prod system DB and SE entries.
* Latest production output is covered. A few problematic files found.
* After v5r4p2 deployment, no lost file has been observed.

## Development matters

### BelleDIRAC Certification - v5r5 with DIRAC v7r2

* After deployment of v5r5 in the certification server, rDDM fails with authentication issue.
* May be related to mismatch between v7r1 and v7r2.
* Contacting DDM experts offline.
* Q: Are B2Monitoring plots working?
* A (Miyake-san): Not working. All old plots must be removed. Asking Hayasaka-san which table to delete.

### BelleDIRAC migration to DIRAC:v7r3

* Waiting for v7r2 in the certification server to move on with v7r3.
* Q: Update of gfal can be done with this migration?
* A (Miyake-san): DIRACOS is not supported (what about DIRACOS2?)

### Elastic Search account proposal

* Kibana might be opened for DP experts/shifters, and even normal users.

* Default “kibana” account is being migrated to “kibana_system”, thus
apparently should not be used for that purpose. A new account must be used.
* Two roles proposed: b2kibana_admin (can develop dashboard on Kibana) and b2kibana_user (can use the dashboard).

* Kibana at BNL available for tests (VM 06).

* Q: How to use Kibana for users? Talking about roles before use cases is odd.
* A (Miyake-san): Asking feedback from Anil. Also, most recent DIRAC stores monitoring information in ES, providing dashboards with Kibana.

### Open Pull requests

* Without validating current DIRAC v7r2, we cannot add new features.

* Q: Can we go ahead with the migration and proceed after with a only-client patch?

### DIRAC devs meeting

* Last week and this week is hackathon. No news.