# Synapse Seg Project ## switch to HarrisLabGPU ```ssh khlab``` Only work in a tmux ```tmux``` detach wiht control + b then d reattach with ```tmux a``` tmux ls to see active tmux end tmux with ```exit``` ### commands: cds get into science directory cdd get into data directory ## General Tips 1. compressing a zarr to send over email ``` tar -czvf compressed.tar.gz <location of zarr> ``` ssh: getting into dif machine terminal ## Installing miniconda3 For mac: ``` wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh ``` ``` chmod +x Miniconda3-latest-MacOSX-arm64.sh ``` ``` ./Miniconda.... ``` For khlabgpu: ``` wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh ``` ## Uninstalling minconda3 ``` pip install anaconda-clean anaconda-clean ``` ``` rm -r Miniconda3 ``` ## Downloading napari 1. Create + activate new environment ###### these can apply to creating any env, just replace napari with your env name ``` conda create -y -n napari -c conda-forge python=3.9 conda activate napari conda create -n autoseg python=3.9 conda activate autoseg conda install pytorch pytorch-cuda=11.7 boost jupyter -c pytorch -c nvidia ``` :::info :bulb: **After creating + activating a new environment, download all modules in Notes w Vijay doc** :::spoiler Installs pip install cython zarr matplotlib mahotas pip install git+https://github.com/funkelab/funlib.geometry.git pip install git+https://github.com/funkelab/funlib.persistence.git pip install git+https://github.com/funkelab/daisy.git pip install git+https://github.com/funkey/gunpowder.git pip install git+https://github.com/funkelab/funlib.math.git pip install git+https://github.com/funkelab/funlib.evaluate.git pip install git+https://github.com/funkelab/funlib.learn.torch.git pip install git+https://github.com/htem/waterz@hms pip install git+https://github.com/funkelab/funlib.segment.git pip install git+https://github.com/funkelab/lsd.git pip install neuroglancer cloud-volume pip install tensorboard tensorboardx pip install jsmin #### Need to make some quick changes because there are some bugs in one of the packages you pip install: gunpowder. Open in vim (or your preferred text editor) the following files and make the edits: $CONDA_PREFIX/lib/python3.9/site-packages/gunpowder/nodes/add_affinities.py Line 24 and line 187: change np.int32 to np.uint64 $CONDA_PREFIX/lib/python3.9/site-packages/gunpowder/nodes/random_location.py Line 321: change size() to size (ONLY FOR M1 MAC USERS:) $CONDA_PREFIX/lib/python3.9/site-packages/gunpowder/torch/nodes/train.py (+predict.py) Lines 164 and 165: change from this: self.use_cuda = torch.cuda.is_available() self.device = torch.device("cuda" if self.use_cuda else "cpu") To: self.use_mps = torch.backends.mps.is_available() self.device = torch.device("mps" if self.use_mps else "cpu") ^^Basically change any instance of cuda to mps ::: 2. Download napari If Qt being weird run ```CONFIG+=sdk_no_version_check``` ``` pip install "git+https://github.com/napari/napari.git#egg=napari[all]" ``` or ``` pip install "napari[all]"" ``` or actually hopefuly ``` conda install -c conda-forge napari ``` 3. Activate napari ``` napari ``` ## Painting Synapses :::info [Click to familiarize yourself with neuron structures](https://synapseweb.clm.utexas.edu/atlas) ::: 1. Open raw images in napari ``` Click File -> Open Folder, navigate to raw in apical_crop.zarr, click Open ``` 2. Add Labels layer, rename to labels 3. Draw on synapses a. Press 2 to activate brush setting b. click shuffle colors c. draw on synapses 4. Create labels_mask layer and paint fill the biggest boundary around your painted synapses such that all synapses within this boundary are painted by you 5. Save your painted Labels layers individually into .tif files from napari as 'labels' and 'labels_mask' into your synapse project folder :::info Sections 87, 88, 89 are painted ::: # ORRR ## Painting training data in PyReconstruct Clone the pyrecontruct repository: ```git clone https://www.github.com/SynapseWeb/PyReconstruct PyReconstruct``` Switch to "autoseg-meeting" branch: ```git checkout autoseg-meeting``` Pull updates: ```git pull origin autoseg-meeting``` Hard reset, so you actually use this branch: ```git reset --hard origin/autoseg-meeting``` Activate environment: ```source env/bin/activate``` Install required packages: ```pip install -r src/requirements.txt``` Run PyReconstruct: ```python3 src/PyReconstruct.py``` Open jser file Trace Save trace as zarr ## Preparing images to be trained on 1. Run the tif_labels_to_zarr.py script ``` python tif_labels_to_zarr.py ../labels.tif apical_crop.zarr python tif_labels_to_zarr.py ../labels_mask.tif apical_crop.zarr ``` 2. Check if you can view them in neuroglancer ``` python -i zarr_view.py apical_crop.zarr ``` 3. Run the make_2d.py script ``` python make_2d.py apical_crop.zarr ``` View new 2d zarr ``` python -i view_stacked.py 2d_apical_crop.zarr ``` Convert datasets from uint8 to uint64 (jk maybe dont) ``` python convert_uint8_to_uint64.py 2d_apical_crop.zarr ``` ## Create training environment ``` conda create -y -n train_synapse -c conda-forge python=3.9 ``` Install modules from Notes w Vijay doc ## Train ``` PYTORCH_ENABLE_MPS_FALLBACK=1 python train.py ``` To view loss curve during training: ``` tensorboard --logdir . ``` :::warning tensorflow is weird on macs rn, create tf environment ``` conda create -n tf tensorflow conda activate tf ``` https://docs.anaconda.com/anaconda/user-guide/tasks/tensorflow/ ::: View predicted lables: ``` python -i view.py snapshots/batch_1.zarr ``` Predict on some untrained (83): ``` python predict.py <raw file> <raw ds> <section> ``` :::info :::