Investigating [VBMS Documents not Available in Caseflow](https://github.com/department-of-veterans-affairs/caseflow/issues/14081#) #14081

# Investigating [VBMS Documents not Available in Caseflow](https://github.com/department-of-veterans-affairs/caseflow/issues/14081#) #14081 ###### tags: `Echo` # Part 1 - old Acceptance criteria ------------------- - [ ] Ensure all VBMS documents are loading and displaying in Reader ## Plan - [ ] Identify which documents are not loading - [ ] Why not loading? - [ ] Remedy ## Findings DocCount is shown on Case Details page. Reader's Total is shown in Reader. VBMS and VVA counts are shown in eFolder. Logged-in as VACOLAMD: | Case | DocCount | Reader's Total | VBMS docs | VVA docs | | -------- | -------- | -------------- | --------- | -------- | | 3314680 | 289 | 289 | 266 | 23 | | 3621614 | 138 | 138 | 124 | 14 | | 3765285 | 112 | 112 | 100 | 12 | | dd3c3db8 | 149 | 149 | 129 | 20 | Logged-in as BVAJWHITE (doesn't have access to eFolder!): can see same doc counts. ## [3314680](https://appeals.cf.ds.va.gov/queue/appeals/3314680) CAVC type appeal ~~Why is DocCount is of by 1?~~ Nvmd: refreshing the page hours later shows 289 now. I can see documents with receipt date after Oct 2018. ## [3621614](https://appeals.cf.ds.va.gov/queue/appeals/3621614) Found IHP doc (with tags "IR Back" and "IR HTN") ## [3765285](https://appeals.cf.ds.va.gov/queue/appeals/3765285) Found IHP doc (with tags "SC-CD" and "DIC") ## [dd3c3db8](https://appeals.cf.ds.va.gov/queue/appeals/dd3c3db8-bb37-4ea1-8166-7863118ce5ff) Has 2 IHP docs (4/8/2020 and 12/3/2019) No Vet ID shown in Reader! As BVAESLEBOFF (doesn't have access to eFolder!), able to see all 149 docs in Reader. # Part 2 https://github.com/department-of-veterans-affairs/caseflow/issues/14081#issuecomment-624842390 ```ruby appeal=Appeal.find_by(uuid: "00feb68a-c34a-479b-a654-44bc76282a00") docs=Document.where(file_number: appeal.veteran_file_number) docs.count => 426 # Reader and eFolder say 425 (397 from VBMS + 28 from VVA/LCM) # TODO: why a difference of 1? pp docs.pluck(:created_at, :received_at, :upload_date) # received_at and upload_date are pre-2020 # A good number of docs were created on 23 Mar 2020 pp docs.group_by{|d| d.created_at.beginning_of_day}.map{|k,v| [k,v.size]} [[Mon, 23 Mar 2020 00:00:00 UTC +00:00, 421], [Fri, 24 Apr 2020 00:00:00 UTC +00:00, 2], [Sun, 26 Apr 2020 00:00:00 UTC +00:00, 3]] # Almost all the docs were created on 23 Mar 2020 # Who viewed docs? pp DocumentView.where(document: docs).group_by(&:user_id).map{|k,v| [User.find(k).css_id,v.size]} => [["VACOCARUSA", 53], ["VACOMAHMOM", 402], ["BVAJWHITE", 32], ["BVAJPARKER", 5]] # assigned to case: judge BVAJWHITE, atty VACOMAHMOM users=DocumentView.where(document: docs).group_by(&:user_id).map{|k,v| User.find(k)} # when did users first view files pp DocumentView.where(document: docs).group_by(&:user_id).map{|u,a| [User.find(u).css_id, a.pluck(:first_viewed_at).min, a.pluck(:first_viewed_at).max]} => [["VACOCARUSA", Mon, 23 Mar 2020 15:17:23 UTC +00:00, Fri, 17 Apr 2020 17:40:38 UTC +00:00], ["VACOMAHMOM", Tue, 21 Apr 2020 15:00:12 UTC +00:00, Tue, 12 May 2020 11:23:53 UTC +00:00], ["BVAJWHITE", Tue, 28 Apr 2020 20:24:05 UTC +00:00, Wed, 29 Apr 2020 14:20:12 UTC +00:00], ["BVAJPARKER", Fri, 17 Apr 2020 15:04:33 UTC +00:00, Fri, 17 Apr 2020 16:08:22 UTC +00:00]] # When did atty first view docs? pp DocumentView.where(document: docs, user: User.find_by(css_id: "VACOMAHMOM")).group_by{|d| d.first_viewed_at.beginning_of_day}.map{|k,v| [k,v.size]} =>[[Tue, 21 Apr 2020 00:00:00 UTC +00:00, 303], [Mon, 11 May 2020 00:00:00 UTC +00:00, 71], [Wed, 06 May 2020 00:00:00 UTC +00:00, 21], [Tue, 12 May 2020 00:00:00 UTC +00:00, 1], [Wed, 22 Apr 2020 00:00:00 UTC +00:00, 2], [Mon, 27 Apr 2020 00:00:00 UTC +00:00, 4]] # When were these docs created? pp docs.group_by{|d| d.created_at.beginning_of_day}.map{|k,v| [k,v.size]} => [ [Mon, 23 Mar 2020 00:00:00 UTC +00:00, 421], [Fri, 24 Apr 2020 00:00:00 UTC +00:00, 2], [Sun, 26 Apr 2020 00:00:00 UTC +00:00, 3]] # When did judge first view docs pp DocumentView.where(document: docs, user: User.find_by(css_id: "BVAJWHITE")).group_by{|d| d.first_viewed_at.beginning_of_day}.map{|k,v| [k,v.size]} => [[Wed, 29 Apr 2020 00:00:00 UTC +00:00, 22], [Tue, 28 Apr 2020 00:00:00 UTC +00:00, 10]] appeal.treee :id,:status,:created_at,:assigned_at,:updated_at ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ Appeal 16082 (evidence_submission) ───── │ ID │ STATUS │ CREATED_AT │ ASSIGNED_AT │ UPDATED_AT │ CLOSED_AT │ └── RootTask │ 314464 │ on_hold │ 2019-08-07 15:06:14 UTC │ 2019-08-07 15:06:14 UTC │ 2019-08-07 15:06:14 UTC │ │ ├── DistributionTask │ 314465 │ completed │ 2019-08-07 15:06:14 UTC │ 2019-10-21 00:30:18 UTC │ 2020-04-14 18:49:46 UTC │ 2020-04-14 18:49:46 UTC │ │ └── EvidenceSubmissionWindowTask │ 314466 │ completed │ 2019-08-07 15:06:14 UTC │ 2019-08-07 15:06:14 UTC │ 2019-10-21 00:30:18 UTC │ 2019-10-21 00:30:18 UTC │ ├── JudgeAssignTask │ 826685 │ completed │ 2020-04-14 18:49:46 UTC │ 2020-04-14 18:49:46 UTC │ 2020-04-20 11:40:41 UTC │ 2020-04-20 11:40:41 UTC │ └── JudgeDecisionReviewTask │ 837873 │ assigned │ 2020-04-20 11:40:41 UTC │ 2020-05-12 11:53:57 UTC │ 2020-05-12 11:53:57 UTC │ │ ├── AttorneyTask │ 837874 │ completed │ 2020-04-20 11:40:41 UTC │ 2020-04-20 11:40:41 UTC │ 2020-04-28 09:46:47 UTC │ 2020-04-28 09:46:47 UTC │ └── AttorneyRewriteTask │ 863547 │ completed │ 2020-04-29 14:30:06 UTC │ 2020-04-29 14:30:06 UTC │ 2020-05-12 11:53:57 UTC │ 2020-05-12 11:53:57 UTC │ └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ ``` Timeline: * Mar 23: 421 documents were retrieved by Caseflow * Apr 20 Judge assigns to atty * Apr 21 Atty starts to view 309 docs on these dates: [Tue, 21 Apr 2020 00:00:00 UTC +00:00, 303], [Wed, 22 Apr 2020 00:00:00 UTC +00:00, 2], [Mon, 27 Apr 2020 00:00:00 UTC +00:00, 4]] * 5 more documents were retrieved by Caseflow [Fri, 24 Apr 2020, 2], [Sun, 26 Apr 2020, 3] * Apr 28 Atty completes AttorneyTask * Apr 28 Judge views 32 docs [[Wed, 29 Apr 2020 00:00:00 UTC +00:00, 22], [Tue, 28 Apr 2020 00:00:00 UTC +00:00, 10]] * Apr 29 Atty assigned AttorneyRewriteTask * May 6 Atty views 93 docs for the first time on these dates: [Wed, 06 May 2020 00:00:00 UTC +00:00, 21], [Mon, 11 May 2020 00:00:00 UTC +00:00, 71], [Tue, 12 May 2020 00:00:00 UTC +00:00, 1], * May 12 Atty completed AttorneyRewriteTask 426 docs were created in Caseflow on these dates: ``` [Mon, 23 Mar 2020 00:00:00 UTC +00:00, 421], [Fri, 24 Apr 2020 00:00:00 UTC +00:00, 2], [Sun, 26 Apr 2020 00:00:00 UTC +00:00, 3] ``` These docs are created by DocumentFetcher ([code](https://github.com/department-of-veterans-affairs/caseflow/blob/7e38376d2cda2c0d5c8ffe9a5097c54d89aa6d7c/app/services/document_fetcher.rb#L60)) once they have been retrieved. DocumentFetcher uses 2 document_services (EFolderService and VBMSService [code](https://github.com/department-of-veterans-affairs/caseflow/blob/7e38376d2cda2c0d5c8ffe9a5097c54d89aa6d7c/app/services/document_fetcher.rb#L77)) to do retrieval. ## Investing caching app/services/document_fetcher.rb ```ruby def document_service @document_service ||= use_efolder ? EFolderService : VBMSService end def fetch_documents_from_service! doc_struct = document_service.fetch_documents_for(appeal, RequestStore.store[:current_user]) ``` app/services/external_api/vbms_service.rb:33 ```ruby def self.fetch_documents_for(appeal, _user = nil) ExternalApi::VbmsDocumentsForAppeal.new(file_number: appeal.veteran_file_number).fetch end ``` * app/services/external_api/vbms_request_with_file_number.rb:26 * app/services/external_api/vbms_documents_for_appeal.rb:14 ```ruby > FeatureToggle.enabled?(:vbms_pagination) => true > FeatureToggle.enabled?(:vbms_pagination, user: User.find_by(css_id: "VACOMAHMOM")) => true > FeatureToggle.enabled?(:vbms_pagination, user: User.find_by(css_id: "BVAJWHITE")) => true ``` ```ruby cache_key=ExternalApi::EfolderService.document_count_cache_key(appeal.veteran_file_number) Rails.cache.read(cache_key) => 425 Rails.cache.delete(cache_key) # Refreshed Case Details page # Can also do ExternalApi::EfolderService.fetch_document_count(appeal.veteran_file_number, User.find_by(css_id:"BVAJWHITE")) Rails.cache.read(cache_key) => 425 ``` * EFolder document counts are cached: app/services/external_api/efolder_service.rb:8 * VBMS documents are "cached" (retrieved then saved in S3) if it does not already exist in S3 https://github.com/department-of-veterans-affairs/caseflow/blob/43562ed9e75b14c6f949802521da9df6a4214c2b/app/models/document.rb#L122 A document's pdf is provided by app/controllers/document_controller.rb:22, which finds the Document and call `.serve` which gets the contents from S3. # Part 3 From [related investigation](https://hackmd.io/Dm7cR49cRgycy5g7oHiGwA) * Why 425 vs 426? * 6/2/2020: Now 440. `docs.pluck(:series_id).uniq.size => 440` ```ruby pp docs.group_by{|d| d.created_at.beginning_of_day}.map{|k,v| [k,v.size]} [[Mon, 23 Mar 2020 00:00:00 UTC +00:00, 421], [Fri, 24 Apr 2020 00:00:00 UTC +00:00, 2], [Sun, 26 Apr 2020 00:00:00 UTC +00:00, 3], [Tue, 26 May 2020 00:00:00 UTC +00:00, 9], [Fri, 29 May 2020 00:00:00 UTC +00:00, 5]] ``` Reader UI: * `df=appeal.document_fetcher` takes a while finish * `df.number_of_documents => 439` 13 new docs? * `df.documents.group_by{|d| d.upload_date.beginning_of_day}.map{|k,v| [k,v.size]}.sort` ```ruby [Tue, 19 May 2020 00:00:00 UTC +00:00, 1], [Wed, 20 May 2020 00:00:00 UTC +00:00, 3], [Thu, 21 May 2020 00:00:00 UTC +00:00, 2], [Fri, 22 May 2020 00:00:00 UTC +00:00, 1], [Tue, 26 May 2020 00:00:00 UTC +00:00, 2], [Wed, 27 May 2020 00:00:00 UTC +00:00, 1], [Thu, 28 May 2020 00:00:00 UTC +00:00, 2], [Fri, 29 May 2020 00:00:00 UTC +00:00, 2], [Tue, 02 Jun 2020 00:00:00 UTC +00:00, 1]] df.documents.group_by{|d| d.received_at.beginning_of_day}.map{|k,v| [k,v.size]}.sort [Fri, 24 Apr 2020 00:00:00 UTC +00:00, 2], [Tue, 19 May 2020 00:00:00 UTC +00:00, 2], [Wed, 20 May 2020 00:00:00 UTC +00:00, 6], [Thu, 21 May 2020 00:00:00 UTC +00:00, 1], [Fri, 22 May 2020 00:00:00 UTC +00:00, 1], [Thu, 28 May 2020 00:00:00 UTC +00:00, 1], [Fri, 29 May 2020 00:00:00 UTC +00:00, 3]] ``` ```ruby vbms_idsD=docs.pluck(:vbms_document_id) vbms_idsR=df.documents.pluck(:vbms_document_id) vbms_idsD-vbms_idsR => ["{2605FFFC-C9C7-4EF8-BAB8-E1042CB7A92F}", "{50FB8137-8D01-431E-B71D-55F8A6BC7F09}"] vbms_idsR-vbms_idsD => ["{2B507FFF-0CF2-41DA-92A4-0394D3BBF52A}"] Document.find_by(vbms_document_id: "{2605FFFC-C9C7-4EF8-BAB8-E1042CB7A92F}") => #<Document id: 60464063, ... received_at: "2013-01-21", type: "Notification Letter (e.g. VA 20-8993, VA 21-0290, ...", description: nil, series_id: "{2605FFFC-C9C7-4EF8-BAB8-E1042CB7A92F}", previous_document_version_id: nil, upload_date: "2013-01-26", created_at: "2020-03-23 15:17:14", updated_at: "2020-03-23 15:17:14"> Document.find_by(vbms_document_id: "{50FB8137-8D01-431E-B71D-55F8A6BC7F09}") => #<Document id: 60464064, ... received_at: "2012-01-09", type: "Notification Letter (e.g. VA 20-8993, VA 21-0290, ...", description: nil, series_id: "{50FB8137-8D01-431E-B71D-55F8A6BC7F09}", previous_document_version_id: nil, upload_date: "2012-01-20", created_at: "2020-03-23 15:17:14", updated_at: "2020-03-23 15:17:14"> # in EE Record.find_by(version_id: "{2605FFFC-C9C7-4EF8-BAB8-E1042CB7A92F}") =>nil Record.find_by(version_id: "{50FB8137-8D01-431E-B71D-55F8A6BC7F09}") =>nil rec=Record.find_by(version_id: "{2B507FFF-0CF2-41DA-92A4-0394D3BBF52A}") => #<Record id: 1810493489, manifest_source_id: 675867, status: "initialized", ... mime_type: "application/pdf", received_at: "2013-01-21 00:00:00", type_description: "Notification Letter (e.g. VA 20-8993, VA 21-0290, ...", type_id: "184", size: nil, jro: nil, source: "VVABFI", created_at: "2020-06-03 03:49:06", updated_at: "2020-06-03 03:49:06", conversion_status: "not_converted", series_id: "{D2F0B814-AB11-470F-8200-3D129A848295}", version: 1, upload_date: "2020-06-02 00:00:00"> Record.select(:source).limit(1000).pluck(:source).uniq => ["SMS", "P8", "D2D", "VVABFI", "Unknown", "SCAN", "VA.gov", "VES", "VBMS", "VBMS-R", "CACI", "ebn", "Rating Decisions", "CAPRI", "QTC", "VDC", "CSRA (RMC-ESS)", "USER", "DPRIS", "Military Services", "Accurint", "PEBLO (IDES)", "Virtual VA", "VACOLS", "VERIS", "Public Contact Box", "NPRC (PIES)", "VHA_CAPRI", "CSRA", "Veteran", "EBN", "Kofax", "LHI", "Capri"] rec.manifest_source => #<ManifestSource id: 675867, manifest_id: 338646, status: "success", name: "VBMS", fetched_at: "2020-06-03 03:49:07", created_at: "2019-01-07 15:22:15", updated_at: "2020-06-03 03:49:07"> rec.manifest_source.records.count => 412 rec.manifest_source.records.pluck(:source).uniq => ["Unknown", "Rating Decisions", "SMS", "VBMS", "Virtual VA", "VVABFI", "LHI", "CACI", "CAPRI", "CSRA", "EBN", "VACOLS", "VDC", "USER", "SCAN", "VBMS-R", "BVA", "SSA", "PCGL", "Veteran", "Accurint", "VRE", "VHA_CAPRI", "ebn"] ``` * 2 `Document` records were created and retrieved but are no longer retrievable by eFolder * eFolder has 1 `Record` that Reader doesn't know about ```ruby rec.manifest rec.manifest.records.size => 439 # Efolder doc_count (used by Queue) dc=DocumentCounter.new(veteran_file_number: rec.file_number) dc.count => 439 ``` * Refreshing Case Details page and Reader, they now say 440 docs. ```ruby # In certification docs=Document.where(file_number: appeal.veteran_file_number) docs.count => 442 vbms_idsR-vbms_idsD => [] # In EE rec.manifest.records.count => 440 dc=DocumentCounter.new(veteran_file_number: rec.file_number) dc.count => 440 ```

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.