---
# System prepended metadata

title: Untitled

---

- [Yong Jae Lee](https://pages.cs.wisc.edu/~yongjaelee/) - Learning to Understand Our Multimodal World with Minimal Supervision
  
	- [LLaVA:  Large Language and Vision Assistant](https://llava-vl.github.io/)
		- Use CLIP and or MS COCO to generate image-caption pairs
		- Use text captions to generate question-answer pairs with LLMs
		- Merge image, question, answer as triplets to train LMM (large multimodal models)
- [Maurits Kaptein](https://www.mauritskaptein.com/) - Scaling Vision-Based Edge AI Solutions: From Prototype to Global Deployment
  
	- Problems with realizing PoC
		- Hardware too expensive
		- Development too expensive
	- [OAAX](https://www.networkoptix.com/blog/2024/03/05/introducing-the-open-ai-accelerator-standard)
- [Heather Courture](https://www.hdcouture.com/) - Data-Efficient and Generalizable: The Domain-Specific Small Vision Model Revolution
  
	- Challenges for domain specific computer vision
		- Unique image modality (Histopathology, drone, microscopy etc.) vs. General modality (ImageNet)
		- Limited data amount (few hundred images)
		- Computer resource constraints: models too large
	- Solution: Domain-Specific Foundation Model
		- Pretraining domain-specific image dataset on domain-specific foundation model
			- Self-supervised learning using unlabeled images
			- Pretext task instead of manual labels
				- Contrastive learning
				- MAE
		- Supervised learning using labeled images
			- Normal finetuning / supervised learning
	- Examples
		- Satellite - land cover classification, 27k images, 80/20 pre-train/test
			- Same model (ResNet18), get similar performances for different contrastive frameworks
			- Different model sizes, found that smaller models sufficient for small training, but larger models benefit from more data.
			- Pretraining data set matters more than model size
		- Histopathology - predict 9 tissue classes, 100k images train, 7180 images for test
			- Challenge: distribution shift
				- color variations from different scanners
				- staining procedures from different labs
			- Solution: simulate color variations with image augmentation
			- Small model with carefully selected datasets gives good enough performance
			- Large models are better but need more resources
	- Best practices
		- Start with another foundation model to shorten pre-training
		- Use a diverse dataset to capture variations within domain
		- Simulate additional variations with augmentation
- [Guy Lavi](https://embeddedvisionsummit.com/2024/speaker/guy-lavi/) - Continual, On-the-Fly Learning through Sequential, Lightweight Optimization
  
	- Techniques of sequential optimization, the lightweight nature, without losing memory
	- Strategy for retraining
	- Continual learning - SOTA
		- A simple naive strategy (a.k.a finetuning) augmented with additional behavior to counteract catastrophic forgetting.
	- Sequential optimization
- [Brian Geisel](https://geisel.software/) - Using Synthetic Data to Train Computer Vision Models
  
	- Why we need synthetic data
- [Vaibhav Ghadiok](https://www.linkedin.com/in/vghadiok) - 10 Commandments for Building a Vision AI Product
  
	- AI compute has increased 80x in efficiency over the past decade
	- Focus on Solving a Real-World Customer Problem
	  logseq.order-list-type:: number
		- Solve a specific problem first before solving it more generically
		- Separate marketing hype from actual problem solving
		- Seek the simplest solution (maybe not AI)
	- Don't Steal or Kill for GPUs
	  logseq.order-list-type:: number
		- Inference - how can you dramatically reduce your data/AI compute needs?
		- Training - maybe not just AI
	- Respect the technology gap
	  logseq.order-list-type:: number
		- Is there a fundamental scientific or engineering innovation needed to build the product?
			- Human-in-the-loop
			- Controlling the environment
		- Researcher vs. practitioner gap
	- Don't be a hero
	  logseq.order-list-type:: number
		- Don't make the problem artificially harder by limiting the sensors/actuators
		- Perception
			- Use a depth/ranging sensor
			- Real-time kinematic positioning (RTK)
			- Multispectral sensors
		- Actuator
			- Use a better gripper
			- touch sensing with vision
			- suction cup
		- Calibration - in-factory and in situ
	- Use priors
	  logseq.order-list-type:: number
		- help from semi-structured environments
	- Embrace multimodality - sensor fusion is good
	  logseq.order-list-type:: number
		- Classical kalman filter
		- nonlinear optimization
		- Multimodal LLMS
	- Optimize for high data quality
	  logseq.order-list-type:: number
		- Data acquisition quality is vital
			- Carefully choose sensors - Image sensors, lens, time synchronization between sensors
		- Quality of training data is critical
			- Hard negatives are good
			- AI thrives on good quality data
	- Choose the right metrics
	  logseq.order-list-type:: number
		- Use more than one metric, not strongly correlated
	- Don't take the name of MM-LLMs in Vain
	  logseq.order-list-type:: number
		- No emergent capabilities
		- Poor in projective and Euclidean geometry
	- Carefully choose AI inference compute
	  logseq.order-list-type:: number
		- Understand total cost of development
		- AI TOPS is not everything, these matter:
			- Heterogenous compute
			- Memory bandwidth
			- Operator support
			- TOPS/W
			- ...
	- Test, continuously learn and adapt
	  logseq.order-list-type:: number
		- Test in the target deployment environment
			- unlike traditional testing, 100% coverage is infeasible
		- Continuously iterate and improve the system
			- current AI is not adaptive
		- Design systems to be debuggable
	- Conclusion:
		- In almost all successful deployments of AI
			- Human in the loop
			- Constraints on the environments
		- Optimize end-to-end - multimodal sensors, AI, compute, hardware
			- Solution is not always more data
		- Don't judge AI capabilities by human analogies.
- [Will Glaser](https://www.linkedin.com/in/willglaser) - Why Amazon Failed and the Future of CV in Retail (Interview)
  
	- What is grabango?
		- Checkout-free shopping
	- Why did you found grabango?
		- Evolution of computer vision
	- 5-10x faster processing per store
	- Profit shrinkage due to criminal activities
	- First to file patents on this area
	- Amazon doing it actually justified the idea
	- Why could you do it when Amazon can't?
		- User experience - marginally better than Amazon
		- Expensive system - 20x more expensive due to sensor fusion
		- Amazon Go stores limited in SKUs (not using pure vision solution)
	- What's the secret sauce in pure vision solution
		- People
		- Reduction of data
		- Cascaded series of algorithms for filtering
			- Better algorithms come in later
		- Free ground truth
			- Cashier labels the data
			- One month training
	- Cameras on-par with iPhone 5
	- Algorithm?
		- Matrix of algorithms, cascaded, parallel
	- Processing and compute
		- A lot of embedded compute
		- Some in-store servers and some cloud
	-
	-
- [Jacob Marks](https://www.linkedin.com/in/jacob-marks) - How Large Language Models are Impacting Computer Vision
  
	- Multimodal and language models are more and more popular in CV community
	- VLMs
		- Represent visual information in semantically meaningful way
		- Train a multimodal model to ingest/interpret these representations
		- VLMs thrive on
			- Open-world knowledge
			- Rich semantic/visual content
		- Challenges
			- Hallucination
			- Edge
			- Robust training
	- LLM-Aided Visual Reasoning
		- Not actually directly processing images
		- Run vision models on visual data
		- Pass results to LLM for analysis and conclusions
		- Preprocessing the prompt before passing it to LLM
		- Agentic Information Flow - meta-learning, planning to use tools
		- VisProg: Compositional Visual Reasoning
			- Visual programming with modules
		- HuggingGPT
			- uses all the models in HuggingFace, decides which tools to use
		- Pros
			- Flexible & Modular
			- Interpretable
			- Training-free (using foundation models)
		- Cons
			- Latency
			- Reliance on Prompt Engineering
			- Robustness & Reliability