# Migration discrepancy ## 1.Background There was a problem with search_indexes migration in srint 80, causing the backend deployment in _Staging_ instance to fail. After some research, it was conluded that the existing migration in the database doesn't match with the branh for spring 80. To better understand the underlying issue, we show below: - show the existing migration history - migration in sprint 80 branch ## 2. Existing migration from DB history To get the existing migration, we have to run the following commands in shell: ```python from django.db.migrations.recorder import MigrationRecorder existing_migration = MigrationRecorder.Migration.objects.all() for migration in existing_migration: print(migration) ``` The command output: ``` . .. Migration 0007_ssp_m1_ssp_m2_ssp_m3 for search_indexes Migration 0038_user_access_requested_date for users Migration 0008_auto_20230522_1850 for search_indexes Migration 0002_alter_parsererror_error_type for parsers Migration 0003_auto_20230518_1339 for parsers Migration 0004_parsererror_object_uuid for parsers Migration 0005_auto_20230601_1510 for parsers Migration 0006_alter_parsererror_item_number for parsers Migration 0009_auto_20230525_1959 for search_indexes Migration 0010_add_tmp_uuid for search_indexes Migration 0039_alter_user_options for users Migration 0011_gen_uuid for search_indexes Migration 0012_set_uuid_pk for search_indexes Migration 0013_rename_uuid for search_indexes Migration 0014_auto_20230707_1952 for search_indexes Migration 0006_auto_20230726_1448 for parsers Migration 0015_auto_20230724_1830 for search_indexes Migration 0016_auto_20230803_1721 for search_indexes Migration 0006_auto_20230810_1500 for parsers ``` As shown above, the latest migration for _search_indexes_ app are: ``` Migration 0014_auto_20230707_1952 for search_indexes Migration 0015_auto_20230724_1830 for search_indexes Migration 0016_auto_20230803_1721 for search_indexes ``` ## 3. Sprint 80 migrations The _HHS_ [sprint 80](https://github.com/HHS/TANF-app/tree/main/tdrs-backend/tdpservice/search_indexes/migrations) have the following three migrations (only latest ones are shown to save space) for _search_indexes_: ``` . .. 0015_auto_20230810_1500.py 0016_auto_20230817_1635.py 0017_auto_20230804_1935.py 0018_auto_20230816_1917.py ``` ## 4. Problem Note that the last existing migration in the staging instance is `0016_auto_20230803_1721` but the deployment (master branch in HHS) deleted that migration and added another migration: `0016_auto_20230817_1635.py`. This causes migration descrepancy between _the running_ and _the outstanding migration_. ### Missing migration As can be seen in section 2 and 3 above, migration `0015_auto_20230724_1830` has been applied to the DB (see the migration from DB history Section.2). This migration has been deleted and changed with `0015_auto_20230810_1500`. #### Problem with Staging When trying to revert the migration back, we will need the migration file in /app so that we can apply them using migrate command. However, the migration in /app directory has `0015_auto_20230810_1500` but we will need the migration `0015_auto_20230724_1830`. I was able to find `0015_auto_20230724_1830` migration in commit `0cb5a9da038c72b8e846f79a5a05bcb94f742cca` using the command: ``` git log --all --full-history -- "*0015_auto_20230724_1830*" ``` ## 5. Solutions ### 5.1. Revert migration One straight forward solution to the problem is to revert existing migrations back to `0015_auto_20230724_1830` using the cmd: ```python= ./manage.py migrate search_indexes 0015_auto_20230724_1830 ``` since the conflicting migrations both are pointing to this migration, then reverting DB state to this migration will salve the problem and we can easily deploy the new master branch. ### 5.2. Revise the migrations Abother solution to the problem is to: 1. delete the conflicting migrations: ```python= 0016_auto_20230817_1635.py 0017_auto_20230804_1935.py 0018_auto_20230816_1917.py ``` 2. then add the migration `0016_auto_20230803_1721` back 3. and create new migrations and merge them Using this method, then the new migration will be based on `0016_auto_20230803_1721` which is existing state of DB. ## 6. Prodcution status After researching the Staging environment, we decided to look into `tdp-backend-prod` to investigate the status of: 1. Migrations that are applied to the database 2. Migrations existing in the code base in prod in ~/app directory ### 6.1 Migration in DB As mentioned above, to list the migration that are existing in DB, we need to run the code in shell (or look directly in the DB) using the code in Section 2. After running the code we got the following migrations: ``` Migration 0002_standardized_field_names for search_indexes Migration 003_auto_20230127_1553 for search_indexes Migration 0008_auto_20230118_1623 for stts Migration 0009_stt_sample for stts Migration 0004_parsererror for search_indexes Migration 0010_alter_stt_stt_code for stts Migration 0005_delete_parsererror for search_indexes Migration 0001_initial for parsers Migration 0036_auto_20230306_1431 for users Migration 0037_auto_20230306_1434 for users Migration 0006_auto_20230403_1957 for search_indexes Migration 0007_ssp_m1_ssp_m2_ssp_m3 for search_indexes Migration 0038_user_access_requested_date for users Migration 0008_auto_20230522_1850 for search_indexes Migration 0002_alter_parsererror_error_type for parsers Migration 0003_auto_20230518_1339 for parsers Migration 0004_parsererror_object_uuid for parsers Migration 0005_auto_20230601_1510 for parsers Migration 0006_alter_parsererror_item_number for parsers Migration 0009_auto_20230525_1959 for search_indexes Migration 0010_add_tmp_uuid for search_indexes Migration 0011_gen_uuid for search_indexes Migration 0012_set_uuid_pk for search_indexes Migration 0013_rename_uuid for search_indexes Migration 0014_auto_20230707_1952 for search_indexes Migration 0039_alter_user_options for users Migration 0006_auto_20230726_1448 for parsers Migration 0015_auto_20230724_1830 for search_indexes Migration 0016_auto_20230803_1721 for search_indexes ``` ### 6.2 Migrations in ~/app The following migrations exists in the ~/app/tdpservice/search_indexes/ directory: ``` 0001_initial.py 0002_standardized_field_names.py 0003_auto_20230127_1553.py 0004_parsererror.py 0005_delete_parsererror.py 0006_auto_20230403_1957.py 0007_ssp_m1_ssp_m2_ssp_m3.py 0008_auto_20230522_1850.py 0009_auto_20230525_1959.py 0010_add_tmp_uuid.py 0011_gen_uuid.py 0012_set_uuid_pk.py 0013_rename_uuid.py 0014_auto_20230707_1952.py 0015_auto_20230724_1830.py 0016_auto_20230803_1721.py ``` ## 7 References http://v3-apidocs.cloudfoundry.org/version/3.147.0/index.html#common-errors https://docs.djangoproject.com/en/4.2/ref/django-admin/#showmigrations