owned this note
owned this note
Published
Linked with GitHub
# Investigating [VBMS Documents not Available in Caseflow](https://github.com/department-of-veterans-affairs/caseflow/issues/14081#) #14081
###### tags: `Echo`
# Part 1 - old
Acceptance criteria
-------------------
- [ ] Ensure all VBMS documents are loading and displaying in Reader
## Plan
- [ ] Identify which documents are not loading
- [ ] Why not loading?
- [ ] Remedy
## Findings
DocCount is shown on Case Details page.
Reader's Total is shown in Reader.
VBMS and VVA counts are shown in eFolder.
Logged-in as VACOLAMD:
| Case | DocCount | Reader's Total | VBMS docs | VVA docs |
| -------- | -------- | -------------- | --------- | -------- |
| 3314680 | 289 | 289 | 266 | 23 |
| 3621614 | 138 | 138 | 124 | 14 |
| 3765285 | 112 | 112 | 100 | 12 |
| dd3c3db8 | 149 | 149 | 129 | 20 |
Logged-in as BVAJWHITE (doesn't have access to eFolder!): can see same doc counts.
## [3314680](https://appeals.cf.ds.va.gov/queue/appeals/3314680)
CAVC type appeal
~~Why is DocCount is of by 1?~~ Nvmd: refreshing the page hours later shows 289 now.
I can see documents with receipt date after Oct 2018.
## [3621614](https://appeals.cf.ds.va.gov/queue/appeals/3621614)
Found IHP doc (with tags "IR Back" and "IR HTN")
## [3765285](https://appeals.cf.ds.va.gov/queue/appeals/3765285)
Found IHP doc (with tags "SC-CD" and "DIC")
## [dd3c3db8](https://appeals.cf.ds.va.gov/queue/appeals/dd3c3db8-bb37-4ea1-8166-7863118ce5ff)
Has 2 IHP docs (4/8/2020 and 12/3/2019)
No Vet ID shown in Reader!
As BVAESLEBOFF (doesn't have access to eFolder!), able to see all 149 docs in Reader.
# Part 2
https://github.com/department-of-veterans-affairs/caseflow/issues/14081#issuecomment-624842390
```ruby
appeal=Appeal.find_by(uuid: "00feb68a-c34a-479b-a654-44bc76282a00")
docs=Document.where(file_number: appeal.veteran_file_number)
docs.count
=> 426
# Reader and eFolder say 425 (397 from VBMS + 28 from VVA/LCM)
# TODO: why a difference of 1?
pp docs.pluck(:created_at, :received_at, :upload_date)
# received_at and upload_date are pre-2020
# A good number of docs were created on 23 Mar 2020
pp docs.group_by{|d| d.created_at.beginning_of_day}.map{|k,v| [k,v.size]}
[[Mon, 23 Mar 2020 00:00:00 UTC +00:00, 421],
[Fri, 24 Apr 2020 00:00:00 UTC +00:00, 2],
[Sun, 26 Apr 2020 00:00:00 UTC +00:00, 3]]
# Almost all the docs were created on 23 Mar 2020
# Who viewed docs?
pp DocumentView.where(document: docs).group_by(&:user_id).map{|k,v| [User.find(k).css_id,v.size]}
=> [["VACOCARUSA", 53], ["VACOMAHMOM", 402], ["BVAJWHITE", 32], ["BVAJPARKER", 5]]
# assigned to case: judge BVAJWHITE, atty VACOMAHMOM
users=DocumentView.where(document: docs).group_by(&:user_id).map{|k,v| User.find(k)}
# when did users first view files
pp DocumentView.where(document: docs).group_by(&:user_id).map{|u,a| [User.find(u).css_id, a.pluck(:first_viewed_at).min, a.pluck(:first_viewed_at).max]}
=> [["VACOCARUSA",
Mon, 23 Mar 2020 15:17:23 UTC +00:00,
Fri, 17 Apr 2020 17:40:38 UTC +00:00],
["VACOMAHMOM",
Tue, 21 Apr 2020 15:00:12 UTC +00:00,
Tue, 12 May 2020 11:23:53 UTC +00:00],
["BVAJWHITE",
Tue, 28 Apr 2020 20:24:05 UTC +00:00,
Wed, 29 Apr 2020 14:20:12 UTC +00:00],
["BVAJPARKER",
Fri, 17 Apr 2020 15:04:33 UTC +00:00,
Fri, 17 Apr 2020 16:08:22 UTC +00:00]]
# When did atty first view docs?
pp DocumentView.where(document: docs, user: User.find_by(css_id: "VACOMAHMOM")).group_by{|d| d.first_viewed_at.beginning_of_day}.map{|k,v| [k,v.size]}
=>[[Tue, 21 Apr 2020 00:00:00 UTC +00:00, 303],
[Mon, 11 May 2020 00:00:00 UTC +00:00, 71],
[Wed, 06 May 2020 00:00:00 UTC +00:00, 21],
[Tue, 12 May 2020 00:00:00 UTC +00:00, 1],
[Wed, 22 Apr 2020 00:00:00 UTC +00:00, 2],
[Mon, 27 Apr 2020 00:00:00 UTC +00:00, 4]]
# When were these docs created?
pp docs.group_by{|d| d.created_at.beginning_of_day}.map{|k,v| [k,v.size]}
=> [
[Mon, 23 Mar 2020 00:00:00 UTC +00:00, 421],
[Fri, 24 Apr 2020 00:00:00 UTC +00:00, 2],
[Sun, 26 Apr 2020 00:00:00 UTC +00:00, 3]]
# When did judge first view docs
pp DocumentView.where(document: docs, user: User.find_by(css_id: "BVAJWHITE")).group_by{|d| d.first_viewed_at.beginning_of_day}.map{|k,v| [k,v.size]}
=>
[[Wed, 29 Apr 2020 00:00:00 UTC +00:00, 22],
[Tue, 28 Apr 2020 00:00:00 UTC +00:00, 10]]
appeal.treee :id,:status,:created_at,:assigned_at,:updated_at
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
Appeal 16082 (evidence_submission) ───── │ ID │ STATUS │ CREATED_AT │ ASSIGNED_AT │ UPDATED_AT │ CLOSED_AT │
└── RootTask │ 314464 │ on_hold │ 2019-08-07 15:06:14 UTC │ 2019-08-07 15:06:14 UTC │ 2019-08-07 15:06:14 UTC │ │
├── DistributionTask │ 314465 │ completed │ 2019-08-07 15:06:14 UTC │ 2019-10-21 00:30:18 UTC │ 2020-04-14 18:49:46 UTC │ 2020-04-14 18:49:46 UTC │
│ └── EvidenceSubmissionWindowTask │ 314466 │ completed │ 2019-08-07 15:06:14 UTC │ 2019-08-07 15:06:14 UTC │ 2019-10-21 00:30:18 UTC │ 2019-10-21 00:30:18 UTC │
├── JudgeAssignTask │ 826685 │ completed │ 2020-04-14 18:49:46 UTC │ 2020-04-14 18:49:46 UTC │ 2020-04-20 11:40:41 UTC │ 2020-04-20 11:40:41 UTC │
└── JudgeDecisionReviewTask │ 837873 │ assigned │ 2020-04-20 11:40:41 UTC │ 2020-05-12 11:53:57 UTC │ 2020-05-12 11:53:57 UTC │ │
├── AttorneyTask │ 837874 │ completed │ 2020-04-20 11:40:41 UTC │ 2020-04-20 11:40:41 UTC │ 2020-04-28 09:46:47 UTC │ 2020-04-28 09:46:47 UTC │
└── AttorneyRewriteTask │ 863547 │ completed │ 2020-04-29 14:30:06 UTC │ 2020-04-29 14:30:06 UTC │ 2020-05-12 11:53:57 UTC │ 2020-05-12 11:53:57 UTC │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
Timeline:
* Mar 23: 421 documents were retrieved by Caseflow
* Apr 20 Judge assigns to atty
* Apr 21 Atty starts to view 309 docs on these dates:
[Tue, 21 Apr 2020 00:00:00 UTC +00:00, 303],
[Wed, 22 Apr 2020 00:00:00 UTC +00:00, 2],
[Mon, 27 Apr 2020 00:00:00 UTC +00:00, 4]]
* 5 more documents were retrieved by Caseflow
[Fri, 24 Apr 2020, 2],
[Sun, 26 Apr 2020, 3]
* Apr 28 Atty completes AttorneyTask
* Apr 28 Judge views 32 docs
[[Wed, 29 Apr 2020 00:00:00 UTC +00:00, 22],
[Tue, 28 Apr 2020 00:00:00 UTC +00:00, 10]]
* Apr 29 Atty assigned AttorneyRewriteTask
* May 6 Atty views 93 docs for the first time on these dates:
[Wed, 06 May 2020 00:00:00 UTC +00:00, 21],
[Mon, 11 May 2020 00:00:00 UTC +00:00, 71],
[Tue, 12 May 2020 00:00:00 UTC +00:00, 1],
* May 12 Atty completed AttorneyRewriteTask
426 docs were created in Caseflow on these dates:
```
[Mon, 23 Mar 2020 00:00:00 UTC +00:00, 421],
[Fri, 24 Apr 2020 00:00:00 UTC +00:00, 2],
[Sun, 26 Apr 2020 00:00:00 UTC +00:00, 3]
```
These docs are created by DocumentFetcher ([code](https://github.com/department-of-veterans-affairs/caseflow/blob/7e38376d2cda2c0d5c8ffe9a5097c54d89aa6d7c/app/services/document_fetcher.rb#L60)) once they have been retrieved. DocumentFetcher uses 2 document_services (EFolderService and VBMSService [code](https://github.com/department-of-veterans-affairs/caseflow/blob/7e38376d2cda2c0d5c8ffe9a5097c54d89aa6d7c/app/services/document_fetcher.rb#L77)) to do retrieval.
## Investing caching
app/services/document_fetcher.rb
```ruby
def document_service
@document_service ||= use_efolder ? EFolderService : VBMSService
end
def fetch_documents_from_service!
doc_struct = document_service.fetch_documents_for(appeal, RequestStore.store[:current_user])
```
app/services/external_api/vbms_service.rb:33
```ruby
def self.fetch_documents_for(appeal, _user = nil)
ExternalApi::VbmsDocumentsForAppeal.new(file_number: appeal.veteran_file_number).fetch
end
```
* app/services/external_api/vbms_request_with_file_number.rb:26
* app/services/external_api/vbms_documents_for_appeal.rb:14
```ruby
> FeatureToggle.enabled?(:vbms_pagination)
=> true
> FeatureToggle.enabled?(:vbms_pagination, user: User.find_by(css_id: "VACOMAHMOM"))
=> true
> FeatureToggle.enabled?(:vbms_pagination, user: User.find_by(css_id: "BVAJWHITE"))
=> true
```
```ruby
cache_key=ExternalApi::EfolderService.document_count_cache_key(appeal.veteran_file_number)
Rails.cache.read(cache_key)
=> 425
Rails.cache.delete(cache_key)
# Refreshed Case Details page
# Can also do ExternalApi::EfolderService.fetch_document_count(appeal.veteran_file_number, User.find_by(css_id:"BVAJWHITE"))
Rails.cache.read(cache_key)
=> 425
```
* EFolder document counts are cached: app/services/external_api/efolder_service.rb:8
* VBMS documents are "cached" (retrieved then saved in S3) if it does not already exist in S3
https://github.com/department-of-veterans-affairs/caseflow/blob/43562ed9e75b14c6f949802521da9df6a4214c2b/app/models/document.rb#L122
A document's pdf is provided by app/controllers/document_controller.rb:22, which finds the Document and call `.serve` which gets the contents from S3.
# Part 3
From [related investigation](https://hackmd.io/Dm7cR49cRgycy5g7oHiGwA)
* Why 425 vs 426?
* 6/2/2020: Now 440. `docs.pluck(:series_id).uniq.size => 440`
```ruby
pp docs.group_by{|d| d.created_at.beginning_of_day}.map{|k,v| [k,v.size]}
[[Mon, 23 Mar 2020 00:00:00 UTC +00:00, 421],
[Fri, 24 Apr 2020 00:00:00 UTC +00:00, 2],
[Sun, 26 Apr 2020 00:00:00 UTC +00:00, 3],
[Tue, 26 May 2020 00:00:00 UTC +00:00, 9],
[Fri, 29 May 2020 00:00:00 UTC +00:00, 5]]
```
Reader UI:
* `df=appeal.document_fetcher` takes a while finish
* `df.number_of_documents => 439` 13 new docs?
* `df.documents.group_by{|d| d.upload_date.beginning_of_day}.map{|k,v| [k,v.size]}.sort`
```ruby
[Tue, 19 May 2020 00:00:00 UTC +00:00, 1],
[Wed, 20 May 2020 00:00:00 UTC +00:00, 3],
[Thu, 21 May 2020 00:00:00 UTC +00:00, 2],
[Fri, 22 May 2020 00:00:00 UTC +00:00, 1],
[Tue, 26 May 2020 00:00:00 UTC +00:00, 2],
[Wed, 27 May 2020 00:00:00 UTC +00:00, 1],
[Thu, 28 May 2020 00:00:00 UTC +00:00, 2],
[Fri, 29 May 2020 00:00:00 UTC +00:00, 2],
[Tue, 02 Jun 2020 00:00:00 UTC +00:00, 1]]
df.documents.group_by{|d| d.received_at.beginning_of_day}.map{|k,v| [k,v.size]}.sort
[Fri, 24 Apr 2020 00:00:00 UTC +00:00, 2],
[Tue, 19 May 2020 00:00:00 UTC +00:00, 2],
[Wed, 20 May 2020 00:00:00 UTC +00:00, 6],
[Thu, 21 May 2020 00:00:00 UTC +00:00, 1],
[Fri, 22 May 2020 00:00:00 UTC +00:00, 1],
[Thu, 28 May 2020 00:00:00 UTC +00:00, 1],
[Fri, 29 May 2020 00:00:00 UTC +00:00, 3]]
```
```ruby
vbms_idsD=docs.pluck(:vbms_document_id)
vbms_idsR=df.documents.pluck(:vbms_document_id)
vbms_idsD-vbms_idsR
=> ["{2605FFFC-C9C7-4EF8-BAB8-E1042CB7A92F}",
"{50FB8137-8D01-431E-B71D-55F8A6BC7F09}"]
vbms_idsR-vbms_idsD
=> ["{2B507FFF-0CF2-41DA-92A4-0394D3BBF52A}"]
Document.find_by(vbms_document_id: "{2605FFFC-C9C7-4EF8-BAB8-E1042CB7A92F}")
=> #<Document id: 60464063, ... received_at: "2013-01-21", type: "Notification Letter (e.g. VA 20-8993, VA 21-0290, ...", description: nil, series_id: "{2605FFFC-C9C7-4EF8-BAB8-E1042CB7A92F}", previous_document_version_id: nil, upload_date: "2013-01-26", created_at: "2020-03-23 15:17:14", updated_at: "2020-03-23 15:17:14">
Document.find_by(vbms_document_id: "{50FB8137-8D01-431E-B71D-55F8A6BC7F09}")
=> #<Document id: 60464064, ... received_at: "2012-01-09", type: "Notification Letter (e.g. VA 20-8993, VA 21-0290, ...", description: nil, series_id: "{50FB8137-8D01-431E-B71D-55F8A6BC7F09}", previous_document_version_id: nil, upload_date: "2012-01-20", created_at: "2020-03-23 15:17:14", updated_at: "2020-03-23 15:17:14">
# in EE
Record.find_by(version_id: "{2605FFFC-C9C7-4EF8-BAB8-E1042CB7A92F}")
=>nil
Record.find_by(version_id: "{50FB8137-8D01-431E-B71D-55F8A6BC7F09}")
=>nil
rec=Record.find_by(version_id: "{2B507FFF-0CF2-41DA-92A4-0394D3BBF52A}")
=> #<Record id: 1810493489, manifest_source_id: 675867, status: "initialized", ... mime_type: "application/pdf", received_at: "2013-01-21 00:00:00", type_description: "Notification Letter (e.g. VA 20-8993, VA 21-0290, ...", type_id: "184", size: nil, jro: nil,
source: "VVABFI", created_at: "2020-06-03 03:49:06", updated_at: "2020-06-03 03:49:06", conversion_status: "not_converted", series_id: "{D2F0B814-AB11-470F-8200-3D129A848295}", version: 1, upload_date: "2020-06-02 00:00:00">
Record.select(:source).limit(1000).pluck(:source).uniq
=> ["SMS", "P8", "D2D", "VVABFI", "Unknown", "SCAN", "VA.gov", "VES", "VBMS", "VBMS-R", "CACI", "ebn", "Rating Decisions", "CAPRI", "QTC", "VDC", "CSRA (RMC-ESS)", "USER", "DPRIS", "Military Services", "Accurint", "PEBLO (IDES)", "Virtual VA", "VACOLS", "VERIS", "Public Contact Box", "NPRC (PIES)", "VHA_CAPRI", "CSRA", "Veteran", "EBN", "Kofax", "LHI", "Capri"]
rec.manifest_source
=> #<ManifestSource id: 675867, manifest_id: 338646, status: "success", name: "VBMS", fetched_at: "2020-06-03 03:49:07", created_at: "2019-01-07 15:22:15", updated_at: "2020-06-03 03:49:07">
rec.manifest_source.records.count
=> 412
rec.manifest_source.records.pluck(:source).uniq
=> ["Unknown", "Rating Decisions", "SMS", "VBMS", "Virtual VA", "VVABFI", "LHI", "CACI", "CAPRI", "CSRA", "EBN", "VACOLS", "VDC", "USER", "SCAN", "VBMS-R", "BVA", "SSA", "PCGL", "Veteran", "Accurint", "VRE", "VHA_CAPRI", "ebn"]
```
* 2 `Document` records were created and retrieved but are no longer retrievable by eFolder
* eFolder has 1 `Record` that Reader doesn't know about
```ruby
rec.manifest
rec.manifest.records.size
=> 439
# Efolder doc_count (used by Queue)
dc=DocumentCounter.new(veteran_file_number: rec.file_number)
dc.count
=> 439
```
* Refreshing Case Details page and Reader, they now say 440 docs.
```ruby
# In certification
docs=Document.where(file_number: appeal.veteran_file_number)
docs.count
=> 442
vbms_idsR-vbms_idsD
=> []
# In EE
rec.manifest.records.count => 440
dc=DocumentCounter.new(veteran_file_number: rec.file_number)
dc.count
=> 440
```